From e5612c8a8d521e59d56cdee9570a4f567c439505 Mon Sep 17 00:00:00 2001 From: hermes Date: Tue, 17 Nov 2026 00:36:27 +0800 Subject: [PATCH] =?UTF-8?q?audit=202026-11-16:=20memory=20stall=2062h54m?= =?UTF-8?q?=20(counts=20unchanged,=20cron=5Fincidents=203->5=20unclosed,?= =?UTF-8?q?=20consolidator=20zero-item=20streak=20120);=20NEW=20incident?= =?UTF-8?q?=20=E2=80=94=2011-15=20plugin=20load-timeout=20storm=20(52=20fa?= =?UTF-8?q?iled=20loads,=20519=20dropped=20registrations=20incl.=20conduit?= =?UTF-8?q?=5Fpush=20474,=2011=20pre=5Ftool=5Fcall=20+=202=20pre=5Fllm=5Fc?= =?UTF-8?q?all=20hook=20timeouts,=2010=20refusals)=20never=20recorded=20by?= =?UTF-8?q?=20the=2011-15=20audit;=20hook=20page=20Recurrence=202=20(load?= =?UTF-8?q?=20average=20removed=20as=20predictor);=20fleet=2029=20jobs,=20?= =?UTF-8?q?11-16=20ledger=20175=20rows=20173/2,=20deliveries=2011/11?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- current-state.md | 36 +++++- ...1-08-pre-tool-call-hook-timeout-cascade.md | 33 ++++- .../2026-11-14-mnemosyne-capture-stall.md | 42 ++++++- .../2026-11-15-plugin-load-timeout-storm.md | 117 ++++++++++++++++++ index.md | 12 +- log.md | 43 +++++++ systems/conduit.md | 17 ++- systems/mnemosyne.md | 6 +- systems/scheduled-tasks.md | 15 ++- 9 files changed, 299 insertions(+), 22 deletions(-) create mode 100644 incidents/2026-11-15-plugin-load-timeout-storm.md diff --git a/current-state.md b/current-state.md index fc5156f..6529e4d 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,44 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-16 -verified_on: 2026-11-16 -review_after: 2026-11-17 +updated: 2026-11-17 +verified_on: 2026-11-17 +review_after: 2026-11-18 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] --- # Current State -Updated: 2026-11-16 +Updated: 2026-11-17 + +## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) + +**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin +load-timeout storm nobody had logged.** + +- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130** + / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed + `alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item + streak **120** cycles. No cause, no fix, `status: active` → + [[incidents/2026-11-14-mnemosyne-capture-stall]] +- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin + loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were + absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and + **10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it + cleared with no fix applied. The companion + [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average + is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and + a *larger* `state.db`). +- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows — + 173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far: + 5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**. +- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** — + nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**, + load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10). +- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at + 11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped. + ## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) diff --git a/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md index f717afd..5518e72 100644 --- a/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +++ b/incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md @@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026 type: incident status: active created: 2026-11-09 -updated: 2026-11-14 -verified_on: 2026-11-14 -review_after: 2026-11-17 +updated: 2026-11-17 +verified_on: 2026-11-17 +review_after: 2026-11-20 confidence: high tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor] sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json] @@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`. fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]). +## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals + +Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit +captured none of this**. Exact `errors.log` counts for 11-15: + +- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` — + **5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one). +- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}` + — hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1 + (121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions + are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx). +- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures. +- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over. + +The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations +(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page, +[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here: +**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class +rather than a mechanism. + +**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average: +the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read +(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger* +`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**; +keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it. + + ## See also - [[systems/conduit]] diff --git a/incidents/2026-11-14-mnemosyne-capture-stall.md b/incidents/2026-11-14-mnemosyne-capture-stall.md index be9f8f8..f990dc5 100644 --- a/incidents/2026-11-14-mnemosyne-capture-stall.md +++ b/incidents/2026-11-14-mnemosyne-capture-stall.md @@ -1,17 +1,17 @@ --- -title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read +title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read type: incident status: active created: 2026-11-15 -updated: 2026-11-16 -verified_on: 2026-11-16 +updated: 2026-11-17 +verified_on: 2026-11-17 review_after: 2026-11-18 confidence: high tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate] sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script] --- -# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST) +# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST) **Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**, and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it @@ -110,6 +110,40 @@ proof of common cause** — but the guard and the capture path share the same ru times out at 600 s belongs next to this stall rather than dismissed. +## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120** + +Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still +total.** + +- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory` + **1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) — + **62 h 54 m** with **zero** writes. +- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`, + `failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`: + + | Incident id | `first_seen_at` | `alerted_at` | + | --- | --- | --- | + | `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` | + | `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` | + + The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per + firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating + on any single row. +- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`, + `failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job** + (00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46). + Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate. +- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742** + at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op + consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at + `2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles + still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side + reason the stall cannot self-heal. +- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the + failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture. + `review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting. + + ## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) - Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. diff --git a/incidents/2026-11-15-plugin-load-timeout-storm.md b/incidents/2026-11-15-plugin-load-timeout-storm.md new file mode 100644 index 0000000..304e459 --- /dev/null +++ b/incidents/2026-11-15-plugin-load-timeout-storm.md @@ -0,0 +1,117 @@ +--- +title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) +type: incident +status: unresolved +created: 2026-11-17 +updated: 2026-11-17 +verified_on: 2026-11-17 +review_after: 2026-11-24 +confidence: high +tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load] +sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json] +--- + +# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15) + +**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was +applied; the storm stopped as abruptly as it started, and the loader budget that produced it is +unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the +11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact +`errors.log` grep, not an estimate. + +## What the log records (2026-11-15) + +- **52** × `Failed to load plugin '': load timed out after 10s (import + register() never returned)` +- **519** × `Plugin '' called register_{tool,hook,skill,platform}() after its load timed out; ignored` +- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` +- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41) +- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}` + +Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold +older lines): + +| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals | +| --- | --- | --- | --- | --- | --- | +| 2026-11-12 | 0 | 0 | 0 | 0 | 0 | +| 2026-11-13 | 0 | 0 | 0 | 0 | 0 | +| 2026-11-14 | 4 | 0 | 1 | 1 | 4 | +| **2026-11-15** | **52** | **519** | **11** | **2** | **10** | +| 2026-11-16 | 0 | 0 | 0 | 0 | 0 | +| 2026-11-17 | 0 | 0 | 0 | 0 | 0 | + +### By plugin (2026-11-15) + +| Plugin | Failed loads | Registrations dropped | +| --- | --- | --- | +| `hermes-hq-push` | 26 | 0 | +| `conduit_push` | 9 | **474** | +| `homeassistant` | 7 | 34 | +| `browser-browser-use` | 3 | 3 | +| `basic` | 2 | 2 | +| `cadu-secrets-vault` | 1 | 4 | +| `telegram-platform` | 1 | 1 | +| `nous` | 1 | 0 | +| `drain` | 1 | 0 | +| `browser-browserbase` | 1 | 1 | +| **Total** | **52** | **519** | + +## The dropped registrations are the more serious half + +A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader +gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's +late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across +**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those +processes the Conduit tools, skills and hooks were **not registered at all**, silently. See +[[systems/conduit]]. + +## Clustering + +These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures +spread over hours 01, 04–11 and 13: + +- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load, + `conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes + `pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls + are refused in the same session (`local-989b3642…`). +- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2 + and `homeassistant` all fail within ~55 s; **118** registrations discarded. +- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each + dropping ~52–56 registrations. +- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures** + in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader + failure alongside it. + +## What this is — and is not + +- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no** + plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a + **saturation symptom** on this host, not a wrong setting. +- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] + recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps + guard also blew its **600 s** budget that day (see + [[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were + exceeded in the same 24 h. +- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under + sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at + 88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading + of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not + predict it**. +- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the + audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin + (re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the + 2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing. + +## Open question + +Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open** +for the caller? Today it fails **open** — the job continues, silently without those tools. A check that +asserts **plugin registration** rather than only job exit status would surface this class; the same idea +applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so +it is Anthony's call). + +## See also + +- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade +- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost +- [[systems/hermes-agent]] — plugin loading, config and timeouts +- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure diff --git a/index.md b/index.md index 8edb0d6..20edfb8 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-16 -verified_on: 2026-11-16 +updated: 2026-11-17 +verified_on: 2026-11-17 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] @@ -12,8 +12,14 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-16. +> model routing and automation. Last updated: 2026-11-17. > +> **2026-11-16 (nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) — memory dead at 62 h 54 m, and a correction to the 11-15 record.** +> The capture stall is unchanged in every respect but the counters: **1130** / **1249** rows, newest write `2026-11-14 09:32:59` AWST, **five** unclosed `alerted` incidents, monitor `failure_streak` 5, consolidator zero-item streak **120** cycles. → [[incidents/2026-11-14-mnemosyne-capture-stall]] +> **New:** the 11-15 window also carried a **plugin load-timeout storm** — **52** failed loads, **519** discarded registrations (`conduit_push` 474, so its tools and hooks were absent for those processes), **11** `pre_tool_call` + **2** `pre_llm_call` hook timeouts and **10** refused tool calls, none of it logged by the 11-15 audit. → [[incidents/2026-11-15-plugin-load-timeout-storm]] +> Cron fleet flat at **29 jobs**; 11-16 ledger 175 rows (173/2), deliveries 11/11. **Zero** load failures or hook timeouts on 11-16 and 11-17. +> + > **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`. > > **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour). diff --git a/log.md b/log.md index 26ced65..7ad4fdd 100644 --- a/log.md +++ b/log.md @@ -1801,3 +1801,46 @@ Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]], **Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`. +## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST) + +**Verdict: 1 new page + 7 page updates, nothing archived, no new system.** + +**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173 +`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`); +deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15 +watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load +6.19–7.48. **No new system, no new concept, no page worth archiving.** + +**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous +audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on +**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded +`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools, +skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in +01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0 +/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is +a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`). + +**Pages:** + +- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the + load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the + open question (fail-open loader; a registration health check is proposed, **not applied**). +- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for + the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak + **5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**. +- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in + two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire + **without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor. +- [[systems/conduit]] — 474 dropped registrations and what they cost. +- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers. +- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up). +- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped. + +**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold +(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and +recorded here so the next audit does not re-litigate it. + +**Still open from earlier windows:** the capture stall has no established cause and no fix applied +(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains +**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**. + diff --git a/systems/conduit.md b/systems/conduit.md index dbf450a..5c22e1d 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,8 +3,8 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-11-09 -verified_on: 2026-11-09 +updated: 2026-11-17 +verified_on: 2026-11-17 review_after: 2026-12-08 confidence: high tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call] @@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] +### 2026-11-15 — 474 registrations dropped across eight failed loads + +`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after +10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called +register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events +(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations, +so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the +caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out +**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts: +[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side: +[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. + + ## Open items - **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call. The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 738542f..61f507a 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-11-16 -verified_on: 2026-11-16 +updated: 2026-11-17 +verified_on: 2026-11-17 review_after: 2026-11-18 confidence: high tags: [memory, mnemosyne, backend] @@ -50,7 +50,7 @@ Primary agent memory store for Hermes. ## Known Issues -**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. → [[incidents/2026-11-14-mnemosyne-capture-stall]] +**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. **Update 2026-11-17 00:27 (third read):** nothing has moved — `working_memory` **1130** / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST, so the stall is **62 h 54 m**; unclosed `alerted` incidents **5** (143 rows total), consolidator zero-item streak **120** cycles, monitor `failure_streak: 5`. → [[incidents/2026-11-14-mnemosyne-capture-stall]] **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 9a30f91..51cc64b 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-16 -verified_on: 2026-11-16 -review_after: 2026-11-17 +updated: 2026-11-17 +verified_on: 2026-11-17 +review_after: 2026-11-18 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -14,6 +14,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db # Scheduled Tasks ## Count +- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29 + jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173 + `completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor* + (`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two + alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard + **600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load + failures and **0** hook timeouts, against 52 and 13 the day before — see + [[incidents/2026-11-15-plugin-load-timeout-storm]]. + - **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit). - **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**