audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0)
This commit is contained in:
+36
-4
@@ -3,16 +3,48 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-16
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-16
|
||||||
review_after: 2026-11-15
|
review_after: 2026-11-17
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-11-15
|
Updated: 2026-11-16
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
||||||
|
|
||||||
|
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
|
||||||
|
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
|
||||||
|
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
|
||||||
|
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
|
||||||
|
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
|
||||||
|
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
|
||||||
|
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
|
||||||
|
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
|
||||||
|
all three alerts read alike. **Do not read it as the stall length.**
|
||||||
|
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
|
||||||
|
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
|
||||||
|
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
|
||||||
|
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
|
||||||
|
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
|
||||||
|
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
|
||||||
|
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
|
||||||
|
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
|
||||||
|
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
|
||||||
|
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
|
||||||
|
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
|
||||||
|
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
|
||||||
|
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
|
||||||
|
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
|
||||||
|
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
|
||||||
|
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
|
||||||
|
human-side for the wiki to have missed.
|
||||||
|
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
|
||||||
|
no-date of 136** pages.
|
||||||
|
|
||||||
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
---
|
---
|
||||||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-11-15
|
created: 2026-11-15
|
||||||
updated: 2026-11-15
|
updated: 2026-11-16
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-16
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
# Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
|
||||||
|
|
||||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||||
@@ -21,6 +21,95 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
|
|||||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||||
the same condition was classified informational — see below).
|
the same condition was classified informational — see below).
|
||||||
|
|
||||||
|
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||||||
|
|
||||||
|
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||||||
|
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||||||
|
|
||||||
|
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||||||
|
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||||||
|
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||||||
|
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||||||
|
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||||||
|
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||||||
|
then both stopped inside the same minute.
|
||||||
|
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||||||
|
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||||||
|
observation, no `last_seen_at` accumulation, no escalation:
|
||||||
|
|
||||||
|
| Incident id | `first_seen_at` | `alerted_at` |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||||||
|
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||||||
|
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||||||
|
|
||||||
|
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||||||
|
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||||||
|
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||||||
|
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||||||
|
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||||||
|
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||||||
|
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||||||
|
**re-opens the existing row** instead of fragmenting.
|
||||||
|
|
||||||
|
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||||||
|
|
||||||
|
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||||||
|
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||||||
|
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||||||
|
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||||||
|
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||||||
|
*conversational* message — not the age of the newest capture.**
|
||||||
|
|
||||||
|
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||||||
|
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||||||
|
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||||||
|
|
||||||
|
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||||||
|
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||||||
|
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||||||
|
|
||||||
|
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||||||
|
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||||||
|
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||||||
|
that is not one.
|
||||||
|
|
||||||
|
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||||||
|
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||||||
|
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||||||
|
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||||||
|
|
||||||
|
### The consolidator is alive and every cycle is a deliberate no-op
|
||||||
|
|
||||||
|
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||||||
|
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||||||
|
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||||||
|
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||||||
|
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||||||
|
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||||||
|
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||||||
|
what would have caught this at 10:04 on 11-14.**
|
||||||
|
|
||||||
|
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||||||
|
|
||||||
|
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||||||
|
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||||||
|
|
||||||
|
```
|
||||||
|
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||||||
|
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||||||
|
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||||||
|
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||||||
|
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||||||
|
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||||||
|
times out at 600 s belongs next to this stall rather than dismissed.
|
||||||
|
|
||||||
|
|
||||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||||
|
|
||||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||||
@@ -103,6 +192,11 @@ the loudest signal available and the only one nothing watches.**
|
|||||||
|
|
||||||
## Impact
|
## Impact
|
||||||
|
|
||||||
|
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||||||
|
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||||||
|
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||||||
|
|
||||||
|
|
||||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||||
window. Recall over that period will answer from nothing.
|
window. Recall over that period will answer from nothing.
|
||||||
@@ -133,7 +227,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
|
|||||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||||||
|
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||||||
|
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||||||
|
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||||||
|
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||||||
|
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-16
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-16
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||||
@@ -12,7 +12,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
|||||||
|
|
||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-11-15.
|
> model routing and automation. Last updated: 2026-11-16.
|
||||||
|
>
|
||||||
|
> **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`.
|
||||||
>
|
>
|
||||||
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
|
||||||
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
|
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
|
||||||
|
|||||||
@@ -1782,3 +1782,22 @@ provider, so the cold-start path is untested. Page → `status: degraded` with a
|
|||||||
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
|
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
|
||||||
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
||||||
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
|
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
|
||||||
|
|
||||||
|
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
|
||||||
|
|
||||||
|
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
|
||||||
|
|
||||||
|
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
|
||||||
|
|
||||||
|
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
|
||||||
|
|
||||||
|
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
|
||||||
|
|
||||||
|
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
|
||||||
|
|
||||||
|
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
|
||||||
|
|
||||||
|
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-16
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-16
|
||||||
review_after: 2026-11-13
|
review_after: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log]
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log]
|
||||||
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
|
||||||
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
||||||
|
|
||||||
@@ -102,6 +102,10 @@ Primary agent memory store for Hermes.
|
|||||||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||||
- Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since
|
- Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since
|
||||||
the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception.
|
the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception.
|
||||||
|
⚠ **2026-11-15 → 11-16: failing on a three-slot streak** — `last_status: error`, `failure_streak`
|
||||||
|
**3** (failed `00:00:53.928`, `12:01:24.719`, `2026-11-16 00:01:33.030`), the fleet's **only**
|
||||||
|
non-clean row. Each failure raises a **new** unclosed `cron_incidents` row for the same condition
|
||||||
|
(signature fragmentation — see the capture-stall incident page).
|
||||||
⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran
|
⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran
|
||||||
**12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 →
|
**12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 →
|
||||||
00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against
|
00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against
|
||||||
@@ -119,6 +123,7 @@ Primary agent memory store for Hermes.
|
|||||||
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
|
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
|
||||||
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||||||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||||
|
⚠ **2026-11-15 11:31:20 slot failed after 17 m 49.8 s** — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`. The `11:45` slot was **skipped** and `12:01:25` completed, so it self-recovered with one slot lost. **New signature for this job** — the 2026-08-19 / 10-16 occurrences had the guard quoting *another* job's error. 1 failure in 97 rows across 11-15 → 11-16.
|
||||||
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
|
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
|
||||||
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
|
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
|
||||||
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
|
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-16
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-16
|
||||||
review_after: 2026-11-15
|
review_after: 2026-11-17
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||||
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
|
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
||||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||||
|
|||||||
Reference in New Issue
Block a user