audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11

This commit is contained in:
hermes
2026-11-17 00:36:27 +08:00
parent 4eee02b9a2
commit e5612c8a8d
9 changed files with 299 additions and 22 deletions
+15 -2
View File
@@ -3,8 +3,8 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-11-09
verified_on: 2026-11-09
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-12-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
@@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
### 2026-11-15 — 474 registrations dropped across eight failed loads
`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after
10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called
register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events
(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations,
so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the
caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out
**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts:
[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side:
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
## Open items
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
+3 -3
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-11-16
verified_on: 2026-11-16
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
confidence: high
tags: [memory, mnemosyne, backend]
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
## Known Issues
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. **Update 2026-11-17 00:27 (third read):** nothing has moved — `working_memory` **1130** / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST, so the stall is **62 h 54 m**; unclosed `alerted` incidents **5** (143 rows total), consolidator zero-item streak **120** cycles, monitor `failure_streak: 5`. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
+12 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-16
verified_on: 2026-11-16
review_after: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-18
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
failures and **0** hook timeouts, against 52 and 13 the day before — see
[[incidents/2026-11-15-plugin-load-timeout-storm]].
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**