nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130
This commit is contained in:
+31
-3
@@ -3,11 +3,11 @@ title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-12-08
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
||||
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
|
||||
---
|
||||
|
||||
@@ -230,7 +230,35 @@ The plugin updated itself in-window and its own ledger records the final step as
|
||||
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
|
||||
2026-11-07.
|
||||
|
||||
## 2026-11-08 13:2x–13:3x AWST — its `pre_tool_call` hook was jamming **every** tool call
|
||||
|
||||
The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push/__init__.py:119`), and for
|
||||
**~18 hours** that callback overran Hermes's **30 s** hook budget and was skipped:
|
||||
|
||||
~/.hermes/logs/errors.log
|
||||
2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
|
||||
... (58 lines total)
|
||||
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
|
||||
|
||||
- **58 timeout lines** in `errors.log` (**12** on 2026-11-07, **46** on 2026-11-08) plus **89**
|
||||
*"skipped after previous timeout or while still running"* suppression lines across the logs. **The last
|
||||
occurrence is 2026-11-08 08:21:03** — nothing since the fix below.
|
||||
- **The plugin's caching was NOT the fault — this was measured, not assumed.** `session_titles.py:21
|
||||
CACHE_SECONDS = 30.0`; timed in-process, a hook call costs **3.1 s cold and 0.00 s warm**. The
|
||||
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**. The timeouts track **load
|
||||
spikes** on the 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` with a **40 MB WAL**
|
||||
checkpointing on the same spindle).
|
||||
- **Fix:** `plugins.hook_callback_timeout` **30 → 120** in `~/.hermes/config.yaml` (2026-11-08 13:35 AWST) —
|
||||
verified live in config, and verified *effective* (no timeout logged after 08:21:03).
|
||||
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **8 h 27 m** and **5 h 59 m 51 s** on its two
|
||||
2026-11-08 slots; *wiki-stocktake-maintenance* failed at 04:36:08 on *"No reply: the turn stopped while a
|
||||
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
|
||||
## Open items
|
||||
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
||||
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
|
||||
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
|
||||
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
|
||||
|
||||
@@ -3,11 +3,11 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory, live-errors-log, live-state-db]
|
||||
---
|
||||
|
||||
# Hermes Agent
|
||||
@@ -17,6 +17,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
|
||||
## Current Status
|
||||
- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
|
||||
- **Hook budget raised: `plugins.hook_callback_timeout: 120`** (built-in default **30 s**), set **2026-11-08 13:3x AWST** in `~/.hermes/config.yaml`. Reason: the `conduit_push` plugin's `pre_tool_call` callback — the **only** `pre_tool_call` registrant on the host — overran the 30 s budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`~/.hermes/logs/errors.log`: 12 lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"* suppression lines), which degraded every tool call to slow-then-suppressed. Verified live in config, and verified effective — **no timeout line after 08:21:03**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
- ⚠ **Session store: `~/.hermes/state.db` is 1.89 GB (1,890,656,256 B) with 2,901 sessions / 197,682 messages and a live 40,157,672 B WAL** (measured 2026-11-09 00:0x AWST). It grew **1.7 GB → 1.89 GB** since 2026-11-06; the 2-core CT460 was at load average 5.89 while it checkpointed. The `state.db.corrupted.20260811_151052` artifact (**624.1 MB**, plus a 32 KB `-shm` and a 0-byte `-wal` sibling) was **deleted** in the 2026-11-08 window; `state.db.recovered.recovery.json` (14,566 B, 2026-08-11) remains. **No retention or checkpoint policy is set** — see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
- **Install method:** git
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
|
||||
|
||||
+13
-4
@@ -3,12 +3,12 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-06
|
||||
verified_on: 2026-11-06
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log]
|
||||
---
|
||||
|
||||
# Mnemosyne
|
||||
@@ -98,7 +98,16 @@ Primary agent memory store for Hermes.
|
||||
## Known Management Jobs
|
||||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception.
|
||||
- Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since
|
||||
the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception.
|
||||
⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran
|
||||
**12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 →
|
||||
00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against
|
||||
105,982 B for the last agent-mode file. Live read at 2026-11-09 00:0x: **working 1030/1030 = 100.0 %,
|
||||
episodic 1192/1192 = 100.0 %** (so the `< 95 %` alert is correctly quiet), 1043 embeddings generated in
|
||||
24 h. The script compares with `awk`, not `bc` — **`bc` is not installed on this host**
|
||||
(`mnemosyne-health-monitor.sh:47`), which is why the old monitor's arithmetic silently defaulted instead
|
||||
of failing loudly.
|
||||
⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its
|
||||
00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission
|
||||
([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned.
|
||||
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
|
||||
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
|
||||
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user