audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0)

This commit is contained in:
Tony0410
2026-11-16 00:32:12 +08:00
parent 402d8546e8
commit 9f80926dda
6 changed files with 180 additions and 22 deletions
+36 -4
View File
@@ -3,16 +3,48 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-15 updated: 2026-11-16
verified_on: 2026-11-15 verified_on: 2026-11-16
review_after: 2026-11-15 review_after: 2026-11-17
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-11-15 Updated: 2026-11-16
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
all three alerts read alike. **Do not read it as the stall length.**
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
human-side for the wiki to have missed.
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
no-date of 136** pages.
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) ## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
+107 -8
View File
@@ -1,17 +1,17 @@
--- ---
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15 title: Mnemosyne capture stalled for 38+ hours (2026-11-14 09:33 AWST onward) — three unclosed alerts, freeze total on re-read
type: incident type: incident
status: active status: active
created: 2026-11-15 created: 2026-11-15
updated: 2026-11-15 updated: 2026-11-16
verified_on: 2026-11-15 verified_on: 2026-11-16
review_after: 2026-11-17 review_after: 2026-11-18
confidence: high confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident] tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db] sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
--- ---
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST) # Mnemosyne capture stalled for 38+ hours (from 2026-11-14 09:33 AWST)
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**, **Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
@@ -21,6 +21,95 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
the previous morning and simply had not been noticed, because the monitor's own earlier reading of the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below). the same condition was classified informational — see below).
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
then both stopped inside the same minute.
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
observation, no `last_seen_at` accumulation, no escalation:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
call): normalise the value out of the signature — round it, or omit it — so the same condition
**re-opens the existing row** instead of fragmenting.
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
*conversational* message — not the age of the newest capture.**
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
| --- | --- | --- | --- |
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
that is not one.
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
captured before the stall — the 2026-11-14 00:09:12 run logged
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
### The consolidator is alive and every cycle is a deliberate no-op
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
what would have caught this at 10:04 on 11-14.**
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
```
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
```
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
times out at 600 s belongs next to this stall rather than dismissed.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) ## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. - Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
@@ -103,6 +192,11 @@ the loudest signal available and the only one nothing watches.**
## Impact ## Impact
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST** - ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the (the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing. window. Recall over that period will answer from nothing.
@@ -133,7 +227,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked; incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]). the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check). - ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
## Related ## Related
+5 -3
View File
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-15 updated: 2026-11-16
verified_on: 2026-11-15 verified_on: 2026-11-16
confidence: high confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
@@ -12,7 +12,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-15. > model routing and automation. Last updated: 2026-11-16.
>
> **2026-11-15 (nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST) — the memory pipeline is still dead, and the alarm has been understating it.** [[incidents/2026-11-14-mnemosyne-capture-stall]] is now **38 h 46 m** old and **unchanged on every re-read** — `working_memory` **1130** / `episodic_memory` **1249**, *identical to the 11-15 00:4x and 12:03 reads*, newest rows still the `09:32:58`/`09:32:59` `[MODEL_REFRESH_PROPOSAL]` maintenance writes. `cron_incidents` **1 → 3 unclosed** (138 → **141** rows), all this job, all `failure_type: auth`, each `first_seen_at == last_seen_at` — **new mechanism: the signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so one durable fault mints a fresh row every 12 h and never escalates.** ⚠ **And that ≈4.64 h is not the stall:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so the estate's only alarm has **understated severity on every fire** (11-15 00:03 → 11-14 19:22:30; 11-15 12:03 → 11-15 07:23:25; 11-16 00:03 → 11-15 19:22:59); four 11-15 Pulse turns are uncaptured. The consolidator is alive and every cycle is a **deliberate no-op** (`4696` at `2026-11-15T16:14:14.934306+00:00`, `items_consolidated: 0`, `summary_preview: 'No-op consolidation cycle — timestamp updated for watchdog'`; **~74 consecutive zero-item cycles**) — it bumps its own timestamp, so an age-only gate never notices. **2026-11-15: 168 rows — 165 `completed` / 3 `failed`** (the monitor's `00:00:53` and `12:01:24` slots, plus the watchdog's `11:31:20` slot failing after **17 m 49.8 s** with `runtime deps guard … timed out after 600 seconds`, which skipped the `11:45` slot and self-recovered at `12:01:25`), deliveries **9 / 9 `delivered`** (3 `for_failure`); **2026-11-14 settled 167 / 167 `completed` / 0 `failed`**. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, `failure_streak` **3**). Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**. No interactive session started on 11-15 — all 9 sessions are `source: cron`.
> >
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour). > **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**. > **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
+19
View File
@@ -1782,3 +1782,22 @@ provider, so the cold-start path is untested. Page → `status: degraded` with a
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream). section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]], Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]]. [[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
+9 -4
View File
@@ -3,9 +3,9 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-15 updated: 2026-11-16
verified_on: 2026-11-15 verified_on: 2026-11-16
review_after: 2026-11-13 review_after: 2026-11-18
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log] sources: [live-probe, mnemosyne-db, config, ops-profile-config, live-secrets-scan, live-state-db, live-health-monitor-log]
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
## Known Issues ## Known Issues
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]] **⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). **Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded** — `working_memory` **1130** / `episodic_memory` **1249**, *identical at every read of the stall*, so **0 writes in 38 h 46 m**; newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` and the newest *user-turn* capture is `2026-11-14 09:04:19`. The monitor failed twice more (`12:03:14`, `2026-11-16 00:03:15`) → `cron_incidents` **1 → 3 unclosed** (141 rows total), each a single observation (`first_seen_at == last_seen_at`) because **the signature embeds the elapsed-hours float** (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ **And the `≈4.64 h` in the alert is not a stall duration:** `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message whose session source is not `cron`/`subagent` — the **Pulse/`desktop`** turns, ~6 h cadence — so **the alarm has understated severity on every fire** (mapping table on the incident page). The consolidator's `summary_preview` now reads `No-op consolidation cycle — timestamp updated for watchdog`: it **bumps its own timestamp to satisfy an age test**, so a zero-item streak is invisible to an age-only gate; streak now **~74 cycles** from `2026-11-14T02:04:42+00:00`. **Proposed, none applied:** a zero-item-streak threshold, and normalising the value out of the incident signature. → [[incidents/2026-11-14-mnemosyne-capture-stall]]
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
@@ -102,6 +102,10 @@ Primary agent memory store for Hermes.
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
- Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since - Mnemosyne Health Monitor every 12h — **script**: `mnemosyne-health-monitor.sh`, mode: **no-agent** since
the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception. the 2026-11-08 conversion (job `ea79a2107e36`), reports only on exception.
⚠ **2026-11-15 → 11-16: failing on a three-slot streak** — `last_status: error`, `failure_streak`
**3** (failed `00:00:53.928`, `12:01:24.719`, `2026-11-16 00:01:33.030`), the fleet's **only**
non-clean row. Each failure raises a **new** unclosed `cron_incidents` row for the same condition
(signature fragmentation — see the capture-stall incident page).
⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran ⚠ **The no-agent conversion was verified end-to-end on 2026-11-08:** the 18:00 slot (agent mode) ran
**12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 → **12:00:51 → 18:00:42 = 5 h 59 m 51 s**, and the converted job's first slot ran **2026-11-09 00:00:48 →
00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against 00:02:02 = 73 s**, exit 0, **silent** — `output/ea79a2107e36/2026-11-09_00-02-00.md` is 159 bytes against
@@ -119,6 +123,7 @@ Primary agent memory store for Hermes.
the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.** the 16:37:21 catch-up. **Check `finished_at` before believing a long-running slot failed.**
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent - Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
⚠ **2026-11-15 11:31:20 slot failed after 17 m 49.8 s** — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`. The `11:45` slot was **skipped** and `12:01:25` completed, so it self-recovered with one slot lost. **New signature for this job** — the 2026-08-19 / 10-16 occurrences had the guard quoting *another* job's error. 1 failure in 97 rows across 11-15 → 11-16.
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h, - ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the `b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
+4 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-15 updated: 2026-11-16
verified_on: 2026-11-15 verified_on: 2026-11-16
review_after: 2026-11-15 review_after: 2026-11-17
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth - **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.** consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added, Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,