From e2f1ae7a0b3557929676699bf81b8d80f054f9f1 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Mon, 9 Nov 2026 00:21:01 +0800 Subject: [PATCH] nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130 --- current-state.md | 40 ++++- ...s-at-rest-in-mnemosyne-and-cron-prompts.md | 16 +- ...ron-claim-loss-in-gateway-restart-drain.md | 12 +- ...1-08-pre-tool-call-hook-timeout-cascade.md | 101 +++++++++++++ index.md | 13 +- log.md | 58 +++++++ raw/stocktake/freshness-20261109.txt | 143 ++++++++++++++++++ systems/conduit.md | 34 ++++- systems/hermes-agent.md | 8 +- systems/mnemosyne.md | 17 ++- systems/scheduled-tasks.md | 5 +- systems/skills-index.md | 4 +- 12 files changed, 422 insertions(+), 29 deletions(-) create mode 100644 incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md create mode 100644 raw/stocktake/freshness-20261109.txt diff --git a/current-state.md b/current-state.md index 22e9053..54b3472 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-07 -verified_on: 2026-11-07 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-11-09 confidence: high tags: [system] @@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv --- # Current State -Updated: 2026-11-08 +Updated: 2026-11-09 + +## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST) + +- ⚠ **The `conduit_push` `pre_tool_call` hook was the day's fault, and it is fixed — `plugins.hook_callback_timeout` 30 → 120 s.** + The plugin is the **only** `pre_tool_call` registrant on the host, and its callback overran Hermes's 30 s + hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: 12 + lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"* + suppression lines). Every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST**; + the last timeout line is still 08:21:03, re-verified at audit time. **The plugin's caching was measured, + not assumed, and was innocent** — 3.1 s cold / 0.00 s warm (`session_titles.py:21 CACHE_SECONDS = 30.0`); + the trigger is **load** on the 2-core CT460 (load average **5.89**, `state.db` with a **40 MB WAL**). + → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]], [[systems/conduit]] +- ⚠ **`state.db` is now 1.89 GB** (1,890,656,256 B; **2,901 sessions / 197,682 messages**, live + **40,157,672 B** WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11 + corruption remnant **`state.db.corrupted.20260811_151052` (624.1 MB)** was **deleted** in this window; + `state.db.recovered.recovery.json` (14,566 B) remains, and `mnemosyne.db.corrupted.20260809_120404` + (23.6 MB) was **not** cleaned. → [[systems/hermes-agent]], [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] +- ✅ **The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end.** The 18:00 slot (agent + mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran + **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, **silent**, writing a **159 B** output file against + **105,982 B** for the last agent-mode one. Live coverage **working 1030/1030 = 100.0 %, episodic + 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. → [[systems/mnemosyne]], [[systems/scheduled-tasks]] +- ⚠ **The stocktake failed a third consecutive day, on a new signature.** `1a8e90ea2271` ran + **2026-11-08 03:01:11 → 04:36:08** and died on `RuntimeError: ⚠️ No reply: the turn stopped while a tool + result was still pending and the model produced no follow-up text.` — the previous two red days read + *"lost its durable fire claim ownership"*. **`failure_streak` 3.** *Working hypothesis, not proven:* the + hook above holds a tool result open in exactly the way this error describes. **Unclosed incident rows + 2 → 3**, and all three are silent (`detected`, `alerted_at: null`) — but **the alert path works**: the + *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` at **08:06:54**. + → [[systems/scheduled-tasks]], [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] +- **2026-11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit). Deliveries + **8 / 8 `delivered`**. Fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count, though + `updated_at` moved to `2026-11-09T00:06:50+08:00` with no definition change. Nothing added, retired, + disabled or repinned. → [[systems/scheduled-tasks]] ## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST) diff --git a/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md index 309e0cb..9a92f12 100644 --- a/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +++ b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md @@ -3,8 +3,8 @@ title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and T type: incident status: open created: 2026-11-06 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-11-13 confidence: high tags: [security, credentials, mnemosyne, memory, sessions, cron, incident] @@ -57,7 +57,9 @@ carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shar ### 3. Session transcripts hold all four values -`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks +`~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with +**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was +retentioned), the session store, contains all four values. Measured as 4 MB chunks containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed @@ -72,7 +74,11 @@ becomes a memory row - and it is itself a plaintext at-rest store that deleting **rotation is the only remedy that fully closes it**. - `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key is exposed to anything that reads job definitions. -- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups. +- `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into + snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout + cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. + The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the + 2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched. ## Origin (inferred - not established) @@ -91,7 +97,7 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c 4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or delivery records again. 5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript - verbatim at 1.7 GB. + verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing. ## Verification evidence diff --git a/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md b/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md index 25cdafd..bdd37f2 100644 --- a/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +++ b/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md @@ -3,8 +3,8 @@ title: Two cron runs lost their durable fire claims in a gateway restart drain ( type: incident status: open created: 2026-11-08 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-11-15 confidence: high tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake] @@ -108,6 +108,14 @@ cannot say whether their side effects ran (*"whether side effects ran is unknown - The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or fix the alerting path that let a `detected` row sit silent. + ⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5` + (both first seen 2026-11-07) are unchanged, and a **third** arrived — + `1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08 + failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`. + **Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row + `85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window. + So the gap is specific to the **stocktake's** rows. + → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and whether the 1800 s after-turn wait should refuse to drain long-running cron executions. diff --git a/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md new file mode 100644 index 0000000..f07c099 --- /dev/null +++ b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md @@ -0,0 +1,101 @@ +--- +title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) +type: incident +status: resolved +created: 2026-11-09 +updated: 2026-11-09 +verified_on: 2026-11-09 +review_after: 2026-11-16 +confidence: high +tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor] +sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json] +--- + +# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08) + +**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's +**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip +suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to +*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to +**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 → +14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**. + +## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST) + + grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58 + # 2026-11-07: 12 lines 2026-11-08: 46 lines + # first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863 + grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89 + +The exact pair of lines (`~/.hermes/logs/errors.log`): + + 2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running + 2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping + +**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the +**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place — +`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session: +**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was +therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460 +(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle), +not per-call database opens. + +## Fix applied (2026-11-08 13:35 AWST) + + ~/.hermes/config.yaml + plugins: + hook_callback_timeout: 120 # was the built-in 30 s default + +- **Verified live:** `plugins.hook_callback_timeout = 120`. +- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at + the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last. +- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the + binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching. + +## Consequence — the same signature landed in the cron ledger + +The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed +**`failed`** on a **new terminal signature for that job**: + + RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model + produced no follow-up text. Send `continue` to let it summarize. + +*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this +error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established — +the error string does not name the hook. + +## Impact + +| item | effect | +|---|---| +| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered | +| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row | +| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) | +| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed | +| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation | + +## Verification evidence + + ~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03) + ~/.hermes/config.yaml (plugins.hook_callback_timeout: 120) + ~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration) + ~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0) + ~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows) + +## Open items + +- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load + source behind the spikes. Checkpoint and retention policy are still undecided + ([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). +- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same + fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. +- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch. + +## See also + +- [[systems/conduit]] +- [[systems/hermes-agent]] +- [[systems/scheduled-tasks]] +- [[systems/mnemosyne]] +- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] +- [[current-state]] diff --git a/index.md b/index.md index 35b3c6d..7aacfeb 100644 --- a/index.md +++ b/index.md @@ -3,16 +3,18 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-07 -verified_on: 2026-11-07 +updated: 2026-11-09 +verified_on: 2026-11-09 confidence: high -tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar] +tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-08. +> model routing and automation. Last updated: 2026-11-09. +> +> **2026-11-09 (nightly audit — window 2026-11-08) — the day's fault was a hook, not a schedule, and it is fixed.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`), and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `errors.log` holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while still running"* suppression lines. Every tool call degraded to slow-then-suppressed, which is what stretched the day. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 → 120** in `~/.hermes/config.yaml` — verified live, and verified *effective* (the last timeout line is still 08:21:03, re-checked at audit time). ⚠ **The obvious suspect was measured and acquitted:** the plugin's caching is fine (`session_titles.py:21 CACHE_SECONDS = 30.0`; timed in-process at **3.1 s cold, 0.00 s warm**), so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong** — the trigger is **load** on the 2-core CT460 (`load average 5.89` at 13:27:50, `state.db` checkpointing a **40 MB WAL** on the same spindle) → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday Briefing* took **2 h 09 m 53 s**; and **the stocktake failed a third consecutive day** (`1a8e90ea2271`, 03:01:11 → 04:36:08, `failure_streak: 3`) on a **new terminal signature** — `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` where the previous two red days read *"lost its durable fire claim ownership"*. *Working hypothesis, not proven:* an open tool result is exactly what a suppressed `pre_tool_call` would produce, but the string does not name the hook. **11-08 settled: 169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33); deliveries **8 / 8 `delivered`**; fleet **29 jobs (26 / 3)** — a **sixth** consecutive audit at the same count with nothing added, retired, disabled or repinned, though `updated_at` moved to `2026-11-09T00:06:50+08:00` as a run-state stamp. **Unclosed incident rows 2 → 3**, all three the stocktake's and all silent (`detected`, `alerted_at: null`) — but **the alerting path is not the fault**: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window. ✅ **Two verified resolutions:** the **Mnemosyne Health Monitor's no-agent conversion works** — its first script slot ran **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against **105,982 B** for the last agent-mode run (coverage live at **working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet; the script uses `awk` because **`bc` is not installed on this host**); and the **1.89 GB `state.db`** was re-measured (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL) with the 2026-08-11 remnant **`state.db.corrupted.20260811_151052` (624.1 MB) deleted**. **New page:** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. > > **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. > @@ -67,9 +69,10 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - **RESOLVED (same day) — the hook was the fault, not the schedule.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** → [[systems/conduit]], [[systems/hermes-agent]] - [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. -- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.89 GB session store `state.db`** (**1,890,656,256 B**, **2,901 sessions / 197,682 messages**, live **40 MB** WAL as at 2026-11-09), which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). - [[incidents/2026-11-05-skills-index-generator-divergence]] — **RESOLVED — the skills index flip-flop had two writers.** The repo `wiki-maintain.sh` (302 skills / 68 categories) and the stale `~/.hermes/scripts/wiki-maintain.sh` copy (349 / 26, because it used `Path.rglob()` and counted the parked `.archive/`) were **different scripts**, and SKILL.md pointed the 23:59 nightly audit at the stale one — so each job overwrote the other's page and the wiki oscillated nightly with **no error raised by either**. The stale copy is now a **delegating wrapper**; the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. - [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] diff --git a/log.md b/log.md index ba4d5ad..1775810 100644 --- a/log.md +++ b/log.md @@ -2379,8 +2379,66 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* **after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at **2026-11-08 00:38:29 AWST**. +- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script) run **after** this audit's writes — + `systems/skills-index.md` regenerated (**299 skills / 67 categories**; disk shows 300 `SKILL.md`, one nested + duplicate de-duped by the generator), freshness report `raw/stocktake/freshness-20261109.txt` + (**0 stale / 127 fresh / 3 no-date of 130 pages**), Quartz rebuilt (**130 markdown → 793 files** deployed to + `public/`) and `hermes-wiki-serve.service` (a **user** unit — `systemctl --user`) restarted + **2026-11-09 00:18:30 AWST**. Published pages verified **HTTP 200** at `http://127.0.0.1:9120/` **and** + on the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. - **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.) - **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and the upstream error code only. + +## [2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day +- **New page:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` — **RESOLVED same day.** +- **The finding:** `conduit_push` is the **only** `pre_tool_call` registrant on the host + (`plugins/conduit_push/__init__.py:119`), and its callback overran Hermes's built-in **30 s** hook budget + **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `~/.hermes/logs/errors.log` + holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while + still running"* suppression lines. Every tool call degraded from slow to suppressed. +- **The fix (2026-11-08 13:35 AWST):** `plugins.hook_callback_timeout` **30 → 120** in + `~/.hermes/config.yaml`. **Verified live**, and **verified effective** — the last timeout line is still + 08:21:03, re-checked at audit time. +- **A plausible diagnosis measured and rejected:** the plugin's caching was *not* the fault. Timed + in-process: **3.1 s cold, 0.00 s warm** (`session_titles.py:21 CACHE_SECONDS = 30.0`). The + *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the + 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` checkpointing a **40 MB WAL** on the same + spindle). **Next lever is `state.db` checkpoint/retention discipline, not more caching.** +- **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and + **12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday + Briefing* `ecc1fc13655a` took **2 h 09 m 53 s**; and the stocktake failed a third day. +- **Cron ledger:** 11-08 settled **169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit, + 23:59:33). The single failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08**, on + a **new terminal signature**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still + pending and the model produced no follow-up text.` (the previous two red days read *"lost its durable fire + claim ownership"*). **`failure_streak: 3`.** *Working hypothesis, not proven:* a suppressed `pre_tool_call` + holding a tool result open is exactly what the error describes, but the string does not name the hook. + Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**. Fleet **29 jobs (26 enabled / 3 disabled)** — a + **sixth** consecutive audit at the same count, nothing added, retired, disabled or repinned (fleet + `updated_at` moved to `2026-11-09T00:06:50+08:00`, read as a run-state stamp only). +- **Incidents:** unclosed rows **2 → 3** — a third silent `detected` row for the stocktake + (`1a8e90_a9cad99cb150`, `alerted_at: null`), all three still unreconciled. **The alerting path is not at + fault:** the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and closed `resolved` at + **08:06:54** (its admission-read `database is locked`, cleared when the run completed). +- ✅ **Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works.** The 18:00 slot + (agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran + **2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against + **105,982 B** for the last agent-mode run. Coverage live at **working 1030/1030 = 100.0 %**, **episodic + 1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. The script compares with `awk` because + **`bc` is not installed on this host**. +- ✅ **Verified resolution 2 — `state.db` re-measured and one corruption remnant removed.** Now + **1,890,656,256 B (1.89 GB)**, **2,901 sessions / 197,682 messages**, live **40,157,672 B** WAL. + `state.db.corrupted.20260811_151052` (**624.1 MB** + 32 KB `-shm` + 0 B `-wal`) **deleted** in-window; + `state.db.recovered.recovery.json` (14,566 B) remains and + `mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (**23.6 MB**) was **not** cleaned. +- **Pages updated:** `systems/conduit.md` (the pre-tool-call section, **total**), `systems/hermes-agent.md` + (hook budget + `state.db` size, **total**), `systems/scheduled-tasks.md` (new Count bullet, **total**), + `systems/mnemosyne.md` (health-monitor conversion verified, **total**), `current-state.md` (**total**), + `incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md` (`state.db` 1.7 → 1.89 GB in three + places, re `confident`), `incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md` (open items), + `index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log. +- **No wiki page removed or archived.** No system obsoleted; no decision reversed. +- **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit. diff --git a/raw/stocktake/freshness-20261109.txt b/raw/stocktake/freshness-20261109.txt new file mode 100644 index 0000000..fd54e0d --- /dev/null +++ b/raw/stocktake/freshness-20261109.txt @@ -0,0 +1,143 @@ +# Wiki freshness audit — 2026-11-09 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (127) + +- [ 20d] CONTRIBUTING.md +- [ 1d] agents/hermes-chaos.md +- [ 1d] agents/hermes-production.md +- [ 20d] agents/index.md +- [ 11d] agents/nanobot.md +- [ 20d] agents/open-claw.md +- [ 5d] agents/qwenpaw.md +- [ 20d] agents/zeroclaw.md +- [ 11d] comparisons/camoufox-vs-chrome.md +- [ 20d] comparisons/hermes-vs-nanobot.md +- [ 20d] comparisons/memory-backends.md +- [ 20d] comparisons/model-routers.md +- [ 11d] comparisons/steel-vs-camoufox.md +- [ 20d] concepts/agent-memory.md +- [ 11d] concepts/browser-automation.md +- [ 20d] concepts/model-routing.md +- [ 20d] concepts/persistent-shells.md +- [ 20d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 11d] decisions/index.md +- [ 20d] decisions/keep-openrouter-as-fallback.md +- [ 20d] decisions/persistent-shell-enabled.md +- [ 11d] decisions/rollback-steel-to-camofox.md +- [ 20d] experiments/index.md +- [ 16d] incidents/2026-07-29-mnemosyne-relay.md +- [ 20d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 16d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 2d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 4d] incidents/2026-08-15-skills-index-drift.md +- [ 5d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 15d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 20d] incidents/2026-08-29-gmail-read-state.md +- [ 19d] incidents/2026-09-02-sftpgo-outage.md +- [ 5d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 11d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 14d] incidents/2026-10-03-ct450-hermes-update.md +- [ 14d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 11d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 16d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 8d] incidents/2026-10-09-memory-recall-relevance.md +- [ 3d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 13d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 21d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 13d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 18d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 15d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 14d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 14d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 14d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 12d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 10d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 10d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 10d] incidents/2026-10-29-conduit-voice-silence.md +- [ 9d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 10d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 9d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 8d] incidents/2026-10-31-asana-access-failures.md +- [ 8d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 8d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 7d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 6d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 5d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 3d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 3d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 4d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 0d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 4d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 2d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 1d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 0d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +- [ 0d] index.md +- [ 2d] infrastructure/backups.md +- [ 20d] infrastructure/credentials-map.md +- [ 2d] infrastructure/docker-services.md +- [ 20d] infrastructure/domains-and-tunnels.md +- [ 2d] infrastructure/hosts.md +- [ 2d] infrastructure/network-map.md +- [ 2d] infrastructure/proxmox.md +- [ 2d] infrastructure/tailscale.md +- [ 5d] queries/retained-research-answers.md +- [ 15d] reports/communities-wa-social-media-audit-2026.md +- [ 8d] reports/uber-eats/2026-07.md +- [ 8d] reports/uber-eats/2026-08.md +- [ 8d] reports/uber-eats/2026-09.md +- [ 8d] reports/uber-eats/2026-10.md +- [ 8d] reports/uber-eats/index.md +- [ 20d] runbooks/backup-wiki.md +- [ 20d] runbooks/diagnose-docker.md +- [ 20d] runbooks/gateway-resource-resilience.md +- [ 2d] runbooks/job-radar.md +- [ 20d] runbooks/multiplexer-setup.md +- [ 14d] runbooks/provider-health.md +- [ 11d] runbooks/recover-docker-service.md +- [ 20d] runbooks/restart-browser.md +- [ 20d] runbooks/restart-hermes.md +- [ 20d] runbooks/restore-openrouter.md +- [ 20d] runbooks/test-openrouter.md +- [ 20d] runbooks/troubleshoot-tool-loop.md +- [ 14d] runbooks/update-hermes-safely.md +- [ 1d] systems/browser-backend.md +- [ 20d] systems/cloudflare-tunnel.md +- [ 0d] systems/conduit.md +- [ 14d] systems/current-profile.md +- [ 20d] systems/guanaco.md +- [ 20d] systems/headless-server.md +- [ 0d] systems/hermes-agent.md +- [ 1d] systems/hermes-hq-push.md +- [ 20d] systems/hermes-ui.md +- [ 5d] systems/home-assistant.md +- [ 10d] systems/homepage-dashboard.md +- [ 5d] systems/ignis.md +- [ 14d] systems/memory-backend.md +- [ 7d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 14d] systems/model-providers.md +- [ 5d] systems/nanobot.md +- [ 20d] systems/nextcloud.md +- [ 20d] systems/obsidian.md +- [ 8d] systems/omniroute.md +- [ 1d] systems/opencode-go.md +- [ 20d] systems/openrouter.md +- [ 5d] systems/paseo.md +- [ 21d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 20d] systems/terminal-backend.md +- [ 20d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 127 fresh, 3 no-date of 130 pages \ No newline at end of file diff --git a/systems/conduit.md b/systems/conduit.md index 2fe5ca1..dbf450a 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,11 +3,11 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-12-08 confidence: high -tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] +tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051] --- @@ -230,7 +230,35 @@ The plugin updated itself in-window and its own ledger records the final step as - The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on 2026-11-07. +## 2026-11-08 13:2x–13:3x AWST — its `pre_tool_call` hook was jamming **every** tool call + +The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push/__init__.py:119`), and for +**~18 hours** that callback overran Hermes's **30 s** hook budget and was skipped: + + ~/.hermes/logs/errors.log + 2026-11-07 14:14:03,075 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping + ... (58 lines total) + 2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping + +- **58 timeout lines** in `errors.log` (**12** on 2026-11-07, **46** on 2026-11-08) plus **89** + *"skipped after previous timeout or while still running"* suppression lines across the logs. **The last + occurrence is 2026-11-08 08:21:03** — nothing since the fix below. +- **The plugin's caching was NOT the fault — this was measured, not assumed.** `session_titles.py:21 + CACHE_SECONDS = 30.0`; timed in-process, a hook call costs **3.1 s cold and 0.00 s warm**. The + *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**. The timeouts track **load + spikes** on the 2-core CT460 (`load average 5.89` at 13:27:50; `state.db` with a **40 MB WAL** + checkpointing on the same spindle). +- **Fix:** `plugins.hook_callback_timeout` **30 → 120** in `~/.hermes/config.yaml` (2026-11-08 13:35 AWST) — + verified live in config, and verified *effective* (no timeout logged after 08:21:03). +- **Cost of the fault:** *Mnemosyne Health Monitor* ran **8 h 27 m** and **5 h 59 m 51 s** on its two + 2026-11-08 slots; *wiki-stocktake-maintenance* failed at 04:36:08 on *"No reply: the turn stopped while a + tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**. + → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] + ## Open items +- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call. + The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin. + → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`. - **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome, diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 6d40eac..057bd37 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,11 +3,11 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-09 +verified_on: 2026-11-09 confidence: high tags: [system, hermes, agent, llm] -sources: [live-hermes-version, live-config, live-skill-inventory] +sources: [live-hermes-version, live-config, live-skill-inventory, live-errors-log, live-state-db] --- # Hermes Agent @@ -17,6 +17,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Current Status - **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. +- **Hook budget raised: `plugins.hook_callback_timeout: 120`** (built-in default **30 s**), set **2026-11-08 13:3x AWST** in `~/.hermes/config.yaml`. Reason: the `conduit_push` plugin's `pre_tool_call` callback — the **only** `pre_tool_call` registrant on the host — overran the 30 s budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`~/.hermes/logs/errors.log`: 12 lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"* suppression lines), which degraded every tool call to slow-then-suppressed. Verified live in config, and verified effective — **no timeout line after 08:21:03**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] +- ⚠ **Session store: `~/.hermes/state.db` is 1.89 GB (1,890,656,256 B) with 2,901 sessions / 197,682 messages and a live 40,157,672 B WAL** (measured 2026-11-09 00:0x AWST). It grew **1.7 GB → 1.89 GB** since 2026-11-06; the 2-core CT460 was at load average 5.89 while it checkpointed. The `state.db.corrupted.20260811_151052` artifact (**624.1 MB**, plus a 32 KB `-shm` and a 0-byte `-wal` sibling) was **deleted** in the 2026-11-08 window; `state.db.recovered.recovery.json` (14,566 B, 2026-08-11) remains. **No retention or checkpoint policy is set** — see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Install method:** git - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 3b1de07..a5ea796 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,12 +3,12 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-11-06 -verified_on: 2026-11-06 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-11-13 confidence: high tags: [memory, mnemosyne, backend] -sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db] +sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log] --- # Mnemosyne @@ -98,7 +98,16 @@ Primary agent memory store for Hermes. ## Known Management Jobs - Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent -- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception. +- Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since + the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception. + ⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran + **12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 → + 00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against + 105,982 B for the last agent-mode file. Live read at 2026-11-09 00:0x: **working 1030/1030 = 100.0 %, + episodic 1192/1192 = 100.0 %** (so the `< 95 %` alert is correctly quiet), 1043 embeddings generated in + 24 h. The script compares with `awk`, not `bc` — **`bc` is not installed on this host** + (`mnemosyne-health-monitor.sh:47`), which is why the old monitor's arithmetic silently defaulted instead + of failing loudly. ⚠ **2026-10-30: the monitor was twice flagged and neither flag survived contact with `finished_at`.** Its 00:01:08 slot was auto-paused at 01:04:22 on an *"execution unverified / gateway connection failed"* admission ([[incidents/2026-10-30-mnemosyne-health-monitor-paused]]) — but the slot was **still running** and closed diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 3aefedc..6d02c77 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-09 +verified_on: 2026-11-09 review_after: 2026-11-09 confidence: high tags: [cron, scheduled, automation] @@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned. - **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified diff --git a/systems/skills-index.md b/systems/skills-index.md index b1e12bd..fddb551 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-08 -verified_on: 2026-11-08 +updated: 2026-11-09 +verified_on: 2026-11-09 confidence: high tags: [skills, hermes] sources: []