wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open

- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md
- CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals)
- fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13
  (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows)
- corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery
  question (delivered 00:03:39, for_failure=1)
- updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md,
  infrastructure/docker-services.md
This commit is contained in:
Tony0410
2026-11-15 00:42:01 +08:00
parent 16767ef562
commit 4a6e86948d
8 changed files with 311 additions and 16 deletions
+11 -4
View File
@@ -3,16 +3,23 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
review_after: 2026-11-14 review_after: 2026-11-15
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-11-13 Updated: 2026-11-15
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
- **A third TTS stack landed on CT200** — new page [[systems/phonon-tts]]: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built image). CT200 went **61 → 65 running / 68 → 72 total**; verified live from CT460 (health 200, models list, MP3 returned). ⚠ Everything after **09:03 AWST** on 11-14 is **not attributable to any retained transcript** — see the page's provenance gap before crediting it to a session.
- ⚠ **Open incident: Mnemosyne capture stalled** since **2026-11-14 09:32:59 AWST** (~15 h at this read) — new page [[incidents/2026-11-14-mnemosyne-capture-stall]]. The health monitor escalated only at **2026-11-15 00:03:35** (exit 1, `ea79a2_601037516abd`, `alerted`, unclosed), after logging the same hole as ℹ️ at 12:00 AWST. Root cause not established; the known stringified-config killer is ruled out.
- **Fleet clean apart from the monitor's own alert:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**; `cron_incidents` **0 → 1 unclosed** (the monitor row above). Fleet **29 jobs (26 enabled / 3 disabled)**, **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, streak 1).
- [[infrastructure/docker-services]] CT200 inventory re-verified and updated — **64 non-seeder running / 72 total** (was 61 / 68); the delta is exactly the three TTS containers, no removals.
## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST) ## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST)
@@ -0,0 +1,108 @@
---
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
type: incident
status: active
created: 2026-11-15
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-17
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
---
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below).
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
stopped inside the same minute.
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
(~00:40) **≈15 h 7 m**.
## The alert (and its mislabelling)
```
🚨 Mnemosyne Health Monitor — exception(s):
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
has no working_memory row (last capture 2026-11-14 01:32:59)
```
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
the page records no auth error text. **Observation, not verified diagnosis.**
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below.
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
same hole and calling it benign:
```
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
```
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
**So the fault had ~12 hours of lead time that produced no alarm.**
## Impact
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing.
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
## Not the known config killer
The previously recorded capture-killer on this host is a **stringified list/dict** written into
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
## Open / unverified
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
## Related
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
- [[systems/memory-backend]] — memory backend settings
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
- [[systems/scheduled-tasks]] — the fleet's execution ledger
+6 -2
View File
File diff suppressed because one or more lines are too long
+5 -3
View File
@@ -20,14 +20,16 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u
| Host | Access | Containers running | | Host | Access | Containers running |
|---|---|---| |---|---|---|
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | | CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **65** running / 72 total — main Docker host (**re-verified 2026-11-15**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) | | CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) | | CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented | | CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 | | CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers) ## CT200 live inventory (re-verified 2026-11-15, 65 running / 72 total, excluding 1 transient `worldmonitor-seeder-*` container)
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. `8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, **`phonon-demo`**, **`ptts-openai-server`**, **`ptts-ws-server`**, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
**Re-verified 2026-11-15 00:4x AWST — 61 → 64 running (excluding the seeder), 68 → 72 total, and the whole delta is the TTS stack.** Live `docker ps` on CT200 returns **65 running / 72 total**, which is **64 non-seeder running + 1** transient `worldmonitor-seeder-scheduled`. A set-wise diff of the live names against this page's 2026-11-07 list found **three additions and no removals**, and the arithmetic closes exactly: **61 non-seeder → 64 non-seeder** is the **`phonon-demo`**, **`ptts-openai-server`** and **`ptts-ws-server`** trio — all created 2026-11-14, documented on [[systems/phonon-tts]]. Earlier in this pass a generic `total − prior total` subtraction suggested a fourth, unnamed addition; that was a **comparison artefact** (the 11-07 row's 61 was an explicit *non-seeder* count, today's 65 was a raw `running` count including the seeder) and is **withdrawn** — the name-level diff is exact and accounts for every added container. This pass verified **counts, the TTS trio, and the absence of removals**; it is not a full CT200 audit.
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a **Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200 sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200
+20
View File
@@ -2605,3 +2605,23 @@ hook timeout came back and reopened its incident.**
2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes 2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes
(2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter), (2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter),
`log.md` (this entry). **No page deleted, no decision reversed, no secret written.** `log.md` (this entry). **No page deleted, no decision reversed, no secret written.**
## 2026-11-14 (nightly audit; window 2026-11-13 23:00 → 2026-11-14 23:59 AWST)
**New:** [[systems/phonon-tts]] — the third TTS stack on CT200: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`). Verified live from CT460 — `/health` 200, `/v1/models`, and a POST to `/v1/audio/speech` returning a valid MP3; `:8883` 404s on a plain GET because it is a WebSocket endpoint. `/opt/ptts-voices` holds 7 voices. Page carries the container/timeline table and flags the **provenance gap after 09:03 AWST** — no retained session in this profile covers the 10:31 / 11:00 / 11:43 / 17:54 changes.
**New:** [[incidents/2026-11-14-mnemosyne-capture-stall]] — **open incident**: Mnemosyne has captured nothing since **2026-11-14 09:32:59 AWST** (`working_memory` and `episodic_memory` stop in the same minute; last 8 working writes all 01:01–01:32 UTC). The health monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state `alerted`, unclosed) after ~12 h of lead time in which the same hole was logged **ℹ️ informational** at 12:00 AWST (`capture delay -589.5 min`). ~15 h of desktop/Telegram/cron context uncaptured; the 23:30 sleep cycle consolidated **0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime 2026-11-13 18:03; YAML walk clean). Root cause **not established**.
**Updated:** [[infrastructure/docker-services]] — CT200 re-verified 2026-11-15: **65 running / 72 total** (was 61 / 68 on 11-07), the three TTS containers added to the inventory, **no removals**.
**Updated:** [[current-state]] and [[index]] — dated change log.
**Ledger:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit, started 23:59:37); deliveries **4 / 4 `delivered`**; **`cron_incidents` 0 → 1 unclosed** (the monitor row above, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 jobs (26 enabled / 3 disabled)** — **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13.
**Not changed:** `errors.log` for 11-14 carries only the benign recurring `Browser Use viewing unavailable` plugin warning (~2/hour) plus the already-recorded `pre_tool_call timed out after 120s` at 00:05:40 — see [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] (no new occurrence in the 11-14 body).
## 2026-11-15 00:5x — corrections pass on the 2026-11-14 entry (same audit window)
A second read of the live DBs tightened three claims made in the entry above. All three are applied to the pages; this entry records the corrections rather than leaving two versions in the log.
1. **WITHDRAWN — "one net-added CT200 container unidentified."** The `61 → 65` comparison mixed two different counts: the 2026-11-07 row's **61 was an explicit *non-seeder*** figure while today's **65 was a raw `running`** figure that **includes** the transient `worldmonitor-seeder-scheduled`. A set-wise name diff settles it: live = documented **+ exactly 3** (`phonon-demo`, `ptts-openai-server`, `ptts-ws-server`), **zero removals**, and **64 non-seeder running** against 61 on 11-07. **No unexplained container exists.** [[infrastructure/docker-services]] corrected.
2. **`cron_incidents` went 0 → 1 unclosed** — not "0 unclosed before the 00:03 alert". **138 rows** total; the open one is `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5, `failure_type: auth` — mislabelled). The monitor's **own** job is also the fleet's **single non-clean `last_status`** (`error`, `failure_streak` 1) — the **first red fleet row since 11-13**. [[systems/scheduled-tasks]], [[current-state]] and [[index]] corrected.
3. **RESOLVED (was listed as unverified) — did the 00:03 escalation actually deliver?** Yes. `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — **3 s after** the incident row was raised, **5 s before** `alerted_at`. The **notification path worked; the detection path did not.**
Ledger precision as read at 00:5x: **2026-11-14: 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, 23:59:37); **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (the monitor); deliveries 11-14 **4 / 4 `delivered`** (07:06 / 07:30 / 08:32 / 12:05). Pages also updated in this pass: [[incidents/2026-11-14-mnemosyne-capture-stall]] (delivery question closed, fleet line corrected), [[systems/mnemosyne]] (new ⚠ OPEN block at the head of *Known Issues*, frontmatter to 2026-11-15, and the naive-UTC trap note extended with the monitor's own raw-UTC alert text as a second occurrence of the same trap), [[systems/scheduled-tasks]] (new `### Open 2026-11-15` block under *Known Issues* + the monitor row in the disabled table annotated — **the threshold tests the *sign* of the capture delay, not its magnitude, which is why 12 h of a growing hole raised nothing**).
+5 -3
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-09 updated: 2026-11-15
verified_on: 2026-11-09 verified_on: 2026-11-15
review_after: 2026-11-13 review_after: 2026-11-13
confidence: high confidence: high
tags: [memory, mnemosyne, backend] tags: [memory, mnemosyne, backend]
@@ -50,6 +50,8 @@ Primary agent memory store for Hermes.
## Known Issues ## Known Issues
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below. **Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
@@ -62,7 +64,7 @@ Primary agent memory store for Hermes.
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check. **Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **2026-11-14/15 recurrence, new flavour:** the monitor's own escalation text printed `last capture 2026-11-14 01:32:59` (raw UTC) for a write that landed at **09:32:59 AWST**, so the alert *reads* as a ~23 h hole when the true gap was **14 h 31 m** — the same trap, now inside the monitor's output rather than the audit's reading → [[incidents/2026-11-14-mnemosyne-capture-stall]].
### Vector embedding coverage gap — RESOLVED (2026-08-02) ### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
+120
View File
@@ -0,0 +1,120 @@
---
title: Phonon TTS on CT200 (ptts-openai-server, ptts-ws-server, phonon-demo)
type: system
status: active
created: 2026-11-14
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-12-15
confidence: high
tags: [tts, phonon, ptts, docker, ct200, voice, huggingface, websocket, openai-api, kokoro, edge-tts]
sources: [live-ct200-docker, live-ct200-docker-inspect, live-ct200-files, live-http-probe, live-state-db, live-agent-log, live-mnemosyne-db]
---
# Phonon TTS on CT200
A **third TTS stack** arrived on CT200 on **2026-11-14** — three containers that together form a
self-hosted "ptts" (Phonon TTS) service: an **OpenAI-compatible HTTP API** (`:8881`), a **static
WebAssembly demo UI** (`:8882`), and a **WebSocket TTS server** (`:8883`). It runs alongside the two
voice services that host already carried (`kokoro-tts` `:8880`, `edge-tts` `:5051`).
Page created by the **2026-11-14 nightly audit** from live probes (see *Verification evidence*).
Everything below is a live read unless it says otherwise.
## Containers (live `docker ps` / `docker inspect`, 2026-11-15 ~00:3x AWST)
| Container | Image | Ports (host → container) | Created (UTC) | Last started (UTC) | Restart policy |
|---|---|---|---|---|---|
| `ptts-openai-server` | `xn-ptts-openai-server:v0.7.1` | `8881 → 10200` (**Wyoming also exposed, unmapped**) | `2026-11-14T02:31:56` = **10:31:56 AWST** | `2026-11-14T09:54:11` = **17:54:11 AWST** | `unless-stopped` |
| `phonon-demo` | `nginx:alpine` | `8882 → 80` | `2026-11-14T03:00:11` = **11:00:11 AWST** | `2026-11-14T03:00:11` | `unless-stopped` |
| `ptts-ws-server` | `ptts-ws-server:0.7.1` (**locally built**, 116 MB) | `8883 → 8080` | `2026-11-14T03:43:59` = **11:43:59 AWST** | `2026-11-14T09:54:21` = **17:54:21 AWST** | `unless-stopped` |
Image names/tags for the first row are as reported by `docker ps`; the ws server's image is a
**local build** (`docker image ls ptts-ws-server` → `0.7.1`, no registry prefix), so it is not
re-pullable — **the build inputs are not in the wiki and were not located in this audit.**
## Endpoints (verified live from CT460 over the tailnet, 2026-11-15)
| URL | What answers |
|---|---|
| `http://<CT200 tailnet IP>:8881/health` | health JSON, HTTP 200 |
| `http://<CT200 tailnet IP>:8881/v1/models` | OpenAI-shaped model list → **`gradium/phonon-40m-en`** |
| `http://<CT200 tailnet IP>:8881/v1/audio/speech` | **POST works** — returned a valid MP3 (test voice `nova`) |
| `http://<CT200 tailnet IP>:8882/` | the demo UI — `nginx:alpine`, **19,212 B** body, byte-identical in size to the file below |
| `http://<CT200 tailnet IP>:8883/` | **HTTP 404 on a plain GET** — expected: it is a WebSocket endpoint (`/speech/tts`) |
The `:8881` API is the piece that other services can consume (OpenAI-compatible `/v1/audio/speech`).
## Files on CT200 (owner/paths verified by live `ls -la`)
| Path | Contents |
|---|---|
| `/opt/ptts-openai-server/` | `run.sh` (861 B, `2026-11-14 09:01` local) + `.env` (**47 B, mode 600**, `2026-11-14 09:00` local — holds the HuggingFace token for the gated model; **value not recorded here**) |
| `/opt/phonon-demo/index.html` | the demo page, `2026-11-14 12:30` local; title *"Phonon TTS - WebAssembly Demo"*; talks to **`:8883`** only (no `8881` reference) |
| `/opt/phonon-demo/phonon-tts/` | the WASM/JS assets (`device.js`, `fetch.js`, `hub.js`, `gpu/`, `index.js`, `index.d.ts`, `LICENSE-APACHE`) — i.e. the upstream `phonon-demo` front-end |
| `/opt/ptts-voices/` | **7 entries** (`2026-11-14 17:54` local): six `.safetensors` voices — `anthony.safetensors`, `anthony-clone.safetensors`, `anthony-m.safetensors`, `anthony-november.safetensors`, `elizabeth.safetensors`, `grace.safetensors` — plus a `voices.jsonl` manifest |
The ws server logged **`num_voices=6` at 2026-11-14 04:42:52 UTC (12:42:52 AWST)** and
**`num_voices=7` at 09:54:24 UTC (17:54:24 AWST)** — so the voice set was still being changed that
evening. **Provenance of the voice files is not established**: the naming suggests cloned/derived
voices, but nothing in the retained logs confirms whose they are or how they were produced.
## Deployment timeline (2026-11-14)
| AWST | Event | Source |
|---|---|---|
| 08:35 – 08:45 | First Phonon install session: `ptts-openai-server` brought up on `:8881`; the HuggingFace token returned **401** on the gated model and was replaced | session transcript (`state.db`) |
| 08:52 – 09:03 | Second session: model switched to `gradium/phonon-40m-en`; `/opt/ptts-openai-server/run.sh` (861 B) and its `600` `.env` written at **09:00–09:01** | session transcript + live `ls` |
| 10:31:56 | `ptts-openai-server` **re-created** (2nd recreation; the 09:01 container was not the live one) | live `docker inspect` |
| 11:00:11 | `phonon-demo` (`nginx:alpine`) created → demo UI on `:8882` | live `docker inspect` |
| 11:43:59 | `ptts-ws-server:0.7.1` created → WebSocket TTS on `:8883` | live `docker inspect` |
| 12:30 | `/opt/phonon-demo/index.html` last written | live `ls` mtime |
| 12:42:52 | ws server logs `num_voices=6` | live `docker logs` |
| 17:54:11 / 17:54:21 | **both** ptts servers restarted together; `/opt/ptts-voices` written; ws logs `num_voices=7` | live `docker inspect` + `ls` + `logs` |
⚠ **Provenance gap.** The two retained interactive sessions in `state.db` cover only **08:35–09:03
AWST**. Every change from **10:31 onward** — the demo UI, the ws-server build, the voice files, the
17:54 restart — is **not attributable to any retained transcript**: the only sessions active in
`state.db` between 09:03 and 23:59 on 11-14 are cron runs and a heartbeat "Pulse" session (495
messages, none containing docker/TTS content). **Working hypothesis, not proven: the later work was
done by another estate agent or a session not retained in this profile's store.** Do not treat the
08:35/08:52 sessions as the whole of this deployment.
## Why it is on `8881` and not `8880`
`8880` was already taken by **`kokoro-tts`** on the same host (`ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1`,
declared in the ReadLater compose file → [[infrastructure/docker-services]]), and `edge-tts` sits on
`:5051`. Phonon is therefore the **third** voice backend on CT200 — three TTS stacks on one LXC.
## Operations
```bash
# reach CT200 (the PVE node is 100.96.100.82; the wiki's CT200 row carries the tailnet IP)
ssh root@100.96.100.82 'pct exec 200 -- docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -Ei "ptts|phonon"'
# logs (the ws server is the talkative one)
ssh root@100.96.100.82 'pct exec 200 -- docker logs --tail 40 ptts-ws-server'
# smoke test the OpenAI-compatible path from another host on the tailnet
curl -s -X POST "http://<CT200 tailnet IP>:8881/v1/audio/speech" -H 'Content-Type: application/json' \
-d '{"model":"gradium/phonon-40m-en","input":"test","voice":"nova"}' -o /tmp/probe.mp3 && file /tmp/probe.mp3
```
## Open items / watch
1. **The HuggingFace token is a standing single point of failure.** Both `ptts-openai-server` and
`ptts-ws-server` serve a **gated** model, and the token has already expired once (2026-11-14
08:35–08:45 → replaced). The token lives in `~/.hermes/.env` on CT460 and in
`/opt/ptts-openai-server/.env` (mode 600) on CT200; **whether the ws container reads the same file
was not confirmed in this audit.** A silent expiry would take the whole stack down again.
2. **`ptts-ws-server:0.7.1` is a local-only image** — no build source recorded here.
3. **Attribution gap** for everything after 09:03 AWST (see the timeline note).
4. **Nothing in the estate consumes `:8881` yet**, as far as this audit could see: it was verified by
probe, not by a caller. If it is wired into Hermes TTS later, the provider config belongs in
[[systems/hermes-agent]] / the TTS configuration, not here.
5. The upstream project's own naming is inconsistent (`ptts-*` containers, `phonon-*` model/page, a
"gradium" publisher prefix) — the same stack appears under three names.
## Related
- [[infrastructure/docker-services]] — CT200 inventory and container counts (65 running / 72 total as of 2026-11-15)
- [[systems/mnemosyne]] — the capture stall recorded the same night ([[incidents/2026-11-14-mnemosyne-capture-stall]])
- [[current-state]] — dated change log
+36 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
review_after: 2026-11-14 review_after: 2026-11-15
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` /
1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173,
11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32,
12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status:
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third - **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**, consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned). an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
@@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. | | ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues ## Known Issues
### Open 2026-11-15
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
misreading trap.
### Resolved 2026-11-02 ### Resolved 2026-11-02
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.** - **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**