audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11
This commit is contained in:
+32
-4
@@ -3,16 +3,44 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-16
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-16
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-11-16
|
Updated: 2026-11-17
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
|
||||||
|
|
||||||
|
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
|
||||||
|
load-timeout storm nobody had logged.**
|
||||||
|
|
||||||
|
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
|
||||||
|
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
|
||||||
|
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
|
||||||
|
streak **120** cycles. No cause, no fix, `status: active` →
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
|
||||||
|
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
|
||||||
|
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
|
||||||
|
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
|
||||||
|
cleared with no fix applied. The companion
|
||||||
|
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
|
||||||
|
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
|
||||||
|
a *larger* `state.db`).
|
||||||
|
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
|
||||||
|
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
|
||||||
|
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
|
||||||
|
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
|
||||||
|
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
|
||||||
|
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
|
||||||
|
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
|
||||||
|
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
|
||||||
|
|
||||||
|
|
||||||
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
||||||
|
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-11-09
|
created: 2026-11-09
|
||||||
updated: 2026-11-14
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-14
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||||
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
|
|||||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||||
|
|
||||||
|
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||||
|
|
||||||
|
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||||
|
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||||
|
|
||||||
|
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||||
|
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||||
|
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||||
|
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||||
|
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||||
|
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||||
|
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||||
|
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||||
|
|
||||||
|
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||||
|
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||||
|
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||||
|
rather than a mechanism.
|
||||||
|
|
||||||
|
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||||
|
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||||
|
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||||
|
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||||
|
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||||
|
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
||||||
- [[systems/conduit]]
|
- [[systems/conduit]]
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
---
|
---
|
||||||
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
|
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-11-15
|
created: 2026-11-15
|
||||||
updated: 2026-11-16
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-16
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-18
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
|
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
|
||||||
|
|
||||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||||
@@ -110,6 +110,40 @@ proof of common cause** — but the guard and the capture path share the same ru
|
|||||||
times out at 600 s belongs next to this stall rather than dismissed.
|
times out at 600 s belongs next to this stall rather than dismissed.
|
||||||
|
|
||||||
|
|
||||||
|
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||||||
|
|
||||||
|
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||||||
|
total.**
|
||||||
|
|
||||||
|
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||||||
|
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||||||
|
**62 h 54 m** with **zero** writes.
|
||||||
|
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||||||
|
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||||||
|
|
||||||
|
| Incident id | `first_seen_at` | `alerted_at` |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||||||
|
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||||||
|
|
||||||
|
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||||||
|
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||||||
|
on any single row.
|
||||||
|
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||||||
|
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||||||
|
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||||||
|
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||||||
|
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||||||
|
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||||||
|
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||||||
|
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||||||
|
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||||||
|
reason the stall cannot self-heal.
|
||||||
|
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||||||
|
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||||||
|
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||||
|
|
||||||
|
|
||||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||||
|
|
||||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
---
|
||||||
|
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||||
|
type: incident
|
||||||
|
status: unresolved
|
||||||
|
created: 2026-11-17
|
||||||
|
updated: 2026-11-17
|
||||||
|
verified_on: 2026-11-17
|
||||||
|
review_after: 2026-11-24
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||||
|
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||||
|
|
||||||
|
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||||
|
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||||
|
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||||
|
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||||
|
`errors.log` grep, not an estimate.
|
||||||
|
|
||||||
|
## What the log records (2026-11-15)
|
||||||
|
|
||||||
|
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||||
|
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||||
|
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||||
|
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||||
|
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||||
|
|
||||||
|
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||||
|
older lines):
|
||||||
|
|
||||||
|
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||||
|
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||||
|
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
|
||||||
|
### By plugin (2026-11-15)
|
||||||
|
|
||||||
|
| Plugin | Failed loads | Registrations dropped |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `hermes-hq-push` | 26 | 0 |
|
||||||
|
| `conduit_push` | 9 | **474** |
|
||||||
|
| `homeassistant` | 7 | 34 |
|
||||||
|
| `browser-browser-use` | 3 | 3 |
|
||||||
|
| `basic` | 2 | 2 |
|
||||||
|
| `cadu-secrets-vault` | 1 | 4 |
|
||||||
|
| `telegram-platform` | 1 | 1 |
|
||||||
|
| `nous` | 1 | 0 |
|
||||||
|
| `drain` | 1 | 0 |
|
||||||
|
| `browser-browserbase` | 1 | 1 |
|
||||||
|
| **Total** | **52** | **519** |
|
||||||
|
|
||||||
|
## The dropped registrations are the more serious half
|
||||||
|
|
||||||
|
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||||
|
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||||
|
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||||
|
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||||
|
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||||
|
[[systems/conduit]].
|
||||||
|
|
||||||
|
## Clustering
|
||||||
|
|
||||||
|
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||||
|
spread over hours 01, 04–11 and 13:
|
||||||
|
|
||||||
|
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||||
|
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||||
|
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||||
|
are refused in the same session (`local-989b3642…`).
|
||||||
|
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||||
|
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||||
|
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||||
|
dropping ~52–56 registrations.
|
||||||
|
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||||
|
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||||
|
failure alongside it.
|
||||||
|
|
||||||
|
## What this is — and is not
|
||||||
|
|
||||||
|
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||||
|
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||||
|
**saturation symptom** on this host, not a wrong setting.
|
||||||
|
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||||
|
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||||
|
guard also blew its **600 s** budget that day (see
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||||
|
exceeded in the same 24 h.
|
||||||
|
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||||
|
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||||
|
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||||
|
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||||
|
predict it**.
|
||||||
|
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||||
|
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||||
|
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||||
|
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||||
|
|
||||||
|
## Open question
|
||||||
|
|
||||||
|
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||||
|
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||||
|
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||||
|
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||||
|
it is Anthony's call).
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||||
|
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||||
|
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||||
|
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||||
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-16
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-16
|
verified_on: 2026-11-17
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||||
@@ -12,8 +12,14 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
|
|
||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-11-16.
|
> model routing and automation. Last updated: 2026-11-17.
|
||||||
>
|
>
|
||||||
|
> **2026-11-16 (nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST) — memory dead at 62 h 54 m, and a correction to the 11-15 record.**
|
||||||
|
> The capture stall is unchanged in every respect but the counters: **1130** / **1249** rows, newest write `2026-11-14 09:32:59` AWST, **five** unclosed `alerted` incidents, monitor `failure_streak` 5, consolidator zero-item streak **120** cycles. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
> **New:** the 11-15 window also carried a **plugin load-timeout storm** — **52** failed loads, **519** discarded registrations (`conduit_push` 474, so its tools and hooks were absent for those processes), **11** `pre_tool_call` + **2** `pre_llm_call` hook timeouts and **10** refused tool calls, none of it logged by the 11-15 audit. → [[incidents/2026-11-15-plugin-load-timeout-storm]]
|
||||||
|
> Cron fleet flat at **29 jobs**; 11-16 ledger 175 rows (173/2), deliveries 11/11. **Zero** load failures or hook timeouts on 11-16 and 11-17.
|
||||||
|
>
|
||||||
|
|
||||||
> **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`.
|
> **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`.
|
||||||
>
|
>
|
||||||
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
||||||
|
|||||||
@@ -1801,3 +1801,46 @@ Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
|||||||
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
|
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
|
||||||
|
|
||||||
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
|
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
|
||||||
|
## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST)
|
||||||
|
|
||||||
|
**Verdict: 1 new page + 7 page updates, nothing archived, no new system.**
|
||||||
|
|
||||||
|
**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173
|
||||||
|
`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`);
|
||||||
|
deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15
|
||||||
|
watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load
|
||||||
|
6.19–7.48. **No new system, no new concept, no page worth archiving.**
|
||||||
|
|
||||||
|
**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous
|
||||||
|
audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on
|
||||||
|
**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded
|
||||||
|
`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools,
|
||||||
|
skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in
|
||||||
|
01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0
|
||||||
|
/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is
|
||||||
|
a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`).
|
||||||
|
|
||||||
|
**Pages:**
|
||||||
|
|
||||||
|
- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the
|
||||||
|
load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the
|
||||||
|
open question (fail-open loader; a registration health check is proposed, **not applied**).
|
||||||
|
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for
|
||||||
|
the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak
|
||||||
|
**5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**.
|
||||||
|
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in
|
||||||
|
two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire
|
||||||
|
**without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor.
|
||||||
|
- [[systems/conduit]] — 474 dropped registrations and what they cost.
|
||||||
|
- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers.
|
||||||
|
- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up).
|
||||||
|
- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped.
|
||||||
|
|
||||||
|
**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold
|
||||||
|
(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and
|
||||||
|
recorded here so the next audit does not re-litigate it.
|
||||||
|
|
||||||
|
**Still open from earlier windows:** the capture stall has no established cause and no fix applied
|
||||||
|
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
|
||||||
|
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.
|
||||||
|
|
||||||
|
|||||||
+15
-2
@@ -3,8 +3,8 @@ title: Conduit
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-11-09
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-09
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-12-08
|
review_after: 2026-12-08
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
||||||
@@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push
|
|||||||
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
||||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||||
|
|
||||||
|
### 2026-11-15 — 474 registrations dropped across eight failed loads
|
||||||
|
|
||||||
|
`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after
|
||||||
|
10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called
|
||||||
|
register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events
|
||||||
|
(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations,
|
||||||
|
so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the
|
||||||
|
caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out
|
||||||
|
**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts:
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side:
|
||||||
|
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||||
|
|
||||||
|
|
||||||
## Open items
|
## Open items
|
||||||
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
||||||
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-16
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-16
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-18
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. **Update 2026-11-17 00:27 (third read):** nothing has moved — `working_memory` **1130** / `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST, so the stall is **62 h 54 m**; unclosed `alerted` incidents **5** (143 rows total), consolidator zero-item streak **120** cycles, monitor `failure_streak: 5`. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
|
||||||
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
||||||
|
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-16
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-16
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||||
@@ -14,6 +14,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
|
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
|
||||||
|
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
|
||||||
|
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
|
||||||
|
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
|
||||||
|
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
|
||||||
|
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
|
||||||
|
failures and **0** hook timeouts, against 52 and 13 the day before — see
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]].
|
||||||
|
|
||||||
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
||||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||||
|
|||||||
Reference in New Issue
Block a user