wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
This commit is contained in:
+11
-4
@@ -3,16 +3,23 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-11-13
|
||||
Updated: 2026-11-15
|
||||
|
||||
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
||||
|
||||
- **A third TTS stack landed on CT200** — new page [[systems/phonon-tts]]: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built image). CT200 went **61 → 65 running / 68 → 72 total**; verified live from CT460 (health 200, models list, MP3 returned). ⚠ Everything after **09:03 AWST** on 11-14 is **not attributable to any retained transcript** — see the page's provenance gap before crediting it to a session.
|
||||
- ⚠ **Open incident: Mnemosyne capture stalled** since **2026-11-14 09:32:59 AWST** (~15 h at this read) — new page [[incidents/2026-11-14-mnemosyne-capture-stall]]. The health monitor escalated only at **2026-11-15 00:03:35** (exit 1, `ea79a2_601037516abd`, `alerted`, unclosed), after logging the same hole as ℹ️ at 12:00 AWST. Root cause not established; the known stringified-config killer is ruled out.
|
||||
- **Fleet clean apart from the monitor's own alert:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**; `cron_incidents` **0 → 1 unclosed** (the monitor row above). Fleet **29 jobs (26 enabled / 3 disabled)**, **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, streak 1).
|
||||
- [[infrastructure/docker-services]] CT200 inventory re-verified and updated — **64 non-seeder running / 72 total** (was 61 / 68); the delta is exactly the three TTS containers, no removals.
|
||||
|
||||
## Recent Changes (2026-11-13, nightly audit; live probes 2026-11-13 23:59 – 2026-11-14 00:2x AWST)
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
(exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state **`alerted`**, **unclosed**).
|
||||
|
||||
Found by the **2026-11-14 nightly audit**; not caused by anything in that audit's window (it started
|
||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||
the same condition was classified informational — see below).
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
**2026-11-14 09:32:59 AWST** (the row's own `timestamp` field reads `2026-11-14T09:32:59`).
|
||||
- Newest `episodic_memory.created_at`: **`2026-11-14 01:33:34`** (09:33:34 AWST). **Both** write paths
|
||||
stopped inside the same minute.
|
||||
- Last 8 `working_memory` writes, all on 11-14 (UTC): `01:01:20, 01:02:39, 01:02:40, 01:03:28,
|
||||
01:04:19, 01:32:58, 01:32:59, 01:32:59` — then **nothing for ~15 hours**.
|
||||
- Rows by day (UTC `created_at`): **11-09: 11, 11-10: 10, 11-11: 8, 11-12: 37, 11-13: 14, 11-14: 15**.
|
||||
So the pipeline was *low-volume but alive* before the stall, and the stall is not a normal quiet stretch.
|
||||
- Totals at audit time: **working 1130 / episodic 1249**, DB **27 MB**.
|
||||
- Gap: at the 00:03:35 alert the newest capture was **14 h 30 m 36 s** old; at this audit's read
|
||||
(~00:40) **≈15 h 7 m**.
|
||||
|
||||
## The alert (and its mislabelling)
|
||||
|
||||
```
|
||||
🚨 Mnemosyne Health Monitor — exception(s):
|
||||
|
||||
STALE MEMORY PIPELINE: capture missed: user message 4.64157453 ago (desktop)
|
||||
has no working_memory row (last capture 2026-11-14 01:32:59)
|
||||
```
|
||||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||||
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
|
||||
the page records no auth error text. **Observation, not verified diagnosis.**
|
||||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||||
log line below.
|
||||
|
||||
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
|
||||
|
||||
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
|
||||
same hole and calling it benign:
|
||||
|
||||
```
|
||||
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
|
||||
```
|
||||
|
||||
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
|
||||
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
|
||||
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
|
||||
**So the fault had ~12 hours of lead time that produced no alarm.**
|
||||
|
||||
## Impact
|
||||
|
||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||
window. Recall over that period will answer from nothing.
|
||||
- The **23:30 AWST sleep/consolidation cycle ran and found 0 items** to consolidate
|
||||
(`Last consolidation: 2026-11-14T15:30:51 UTC (0 items)`), so consolidation is *not* compensating.
|
||||
- Nothing else in the estate broke: **2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` /
|
||||
1 `running`** in `executions.db`, deliveries **4 / 4 `delivered`**, and no other cron row went red. This is a
|
||||
**memory-path incident only**. The one exception is the monitor's **own** job: it is now the fleet's
|
||||
**single non-clean row** (`last_status: error`, `failure_streak` 1, previously `ok` / 0) and
|
||||
**`cron_incidents` went 0 → 1 unclosed** (138 rows total). **2026-11-15 so far: 8 rows — 7
|
||||
`completed` / 1 `failed`** (this job), 1 delivery `delivered`.
|
||||
|
||||
## Not the known config killer
|
||||
|
||||
The previously recorded capture-killer on this host is a **stringified list/dict** written into
|
||||
`~/.hermes/config.yaml` by `hermes config set`. Checked and **clean**:
|
||||
- `config.yaml` **mtime 2026-11-13 18:03** — the file was not modified during or before the stall began.
|
||||
- A full YAML walk found **no** scalar that begins with `[` or `{`; `memory.mnemosyne.*` parses as
|
||||
proper lists (`ignore_patterns: []`, `skip_contexts`, `sync_roles`).
|
||||
|
||||
## Open / unverified
|
||||
|
||||
- **Root cause: not established.** No capture-path error lines appear in `agent.log` after 09:33 —
|
||||
the writes simply stop. Candidates to test next: a stuck capture worker or watermark, the
|
||||
`pulse`/heartbeat traffic that ran 12:03–19:22, session-side `memory_enabled` /
|
||||
`mnemosyne.profile_isolation` interaction, or the sleep cycle's `consolidation_claimed_at` state.
|
||||
- ✅ **Resolved — the 00:03 escalation DID deliver.** `deliveries.db` holds exactly one 2026-11-15 row:
|
||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||||
- [[systems/memory-backend]] — memory backend settings
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] — the still-open secrets-at-rest page for the same database
|
||||
- [[systems/scheduled-tasks]] — the fleet's execution ledger
|
||||
@@ -20,14 +20,16 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u
|
||||
|
||||
| Host | Access | Containers running |
|
||||
|---|---|---|
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-07**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **65** running / 72 total — main Docker host (**re-verified 2026-11-15**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) |
|
||||
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-07** — set-wise diff against live `docker ps`: no additions, no removals) |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total (**re-verified 2026-11-07**) — `nostalgic_keldysh`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z. ⚠ **Corrected 2026-11-07:** the enumerated list still opened with `flamboyant_mahavira`, which is **not running** — live `docker ps` on CT333 returns **`nostalgic_keldysh`** (created 2026-11-05 22:15:57, `Up 29 hours`), `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1` (8 running), plus `qdrant` **exited** (9 total) |
|
||||
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
||||
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 |
|
||||
|
||||
## CT200 live inventory (re-verified 2026-11-07, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
## CT200 live inventory (re-verified 2026-11-15, 65 running / 72 total, excluding 1 transient `worldmonitor-seeder-*` container)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, **`phonon-demo`**, **`ptts-openai-server`**, **`ptts-ws-server`**, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
|
||||
**Re-verified 2026-11-15 00:4x AWST — 61 → 64 running (excluding the seeder), 68 → 72 total, and the whole delta is the TTS stack.** Live `docker ps` on CT200 returns **65 running / 72 total**, which is **64 non-seeder running + 1** transient `worldmonitor-seeder-scheduled`. A set-wise diff of the live names against this page's 2026-11-07 list found **three additions and no removals**, and the arithmetic closes exactly: **61 non-seeder → 64 non-seeder** is the **`phonon-demo`**, **`ptts-openai-server`** and **`ptts-ws-server`** trio — all created 2026-11-14, documented on [[systems/phonon-tts]]. Earlier in this pass a generic `total − prior total` subtraction suggested a fourth, unnamed addition; that was a **comparison artefact** (the 11-07 row's 61 was an explicit *non-seeder* count, today's 65 was a raw `running` count including the seeder) and is **withdrawn** — the name-level diff is exact and accounts for every added container. This pass verified **counts, the TTS trio, and the absence of removals**; it is not a full CT200 audit.
|
||||
|
||||
**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a
|
||||
sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200
|
||||
|
||||
@@ -2605,3 +2605,23 @@ hook timeout came back and reopened its incident.**
|
||||
2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes
|
||||
(2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter),
|
||||
`log.md` (this entry). **No page deleted, no decision reversed, no secret written.**
|
||||
|
||||
## 2026-11-14 (nightly audit; window 2026-11-13 23:00 → 2026-11-14 23:59 AWST)
|
||||
|
||||
**New:** [[systems/phonon-tts]] — the third TTS stack on CT200: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`). Verified live from CT460 — `/health` 200, `/v1/models`, and a POST to `/v1/audio/speech` returning a valid MP3; `:8883` 404s on a plain GET because it is a WebSocket endpoint. `/opt/ptts-voices` holds 7 voices. Page carries the container/timeline table and flags the **provenance gap after 09:03 AWST** — no retained session in this profile covers the 10:31 / 11:00 / 11:43 / 17:54 changes.
|
||||
**New:** [[incidents/2026-11-14-mnemosyne-capture-stall]] — **open incident**: Mnemosyne has captured nothing since **2026-11-14 09:32:59 AWST** (`working_memory` and `episodic_memory` stop in the same minute; last 8 working writes all 01:01–01:32 UTC). The health monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state `alerted`, unclosed) after ~12 h of lead time in which the same hole was logged **ℹ️ informational** at 12:00 AWST (`capture delay -589.5 min`). ~15 h of desktop/Telegram/cron context uncaptured; the 23:30 sleep cycle consolidated **0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime 2026-11-13 18:03; YAML walk clean). Root cause **not established**.
|
||||
**Updated:** [[infrastructure/docker-services]] — CT200 re-verified 2026-11-15: **65 running / 72 total** (was 61 / 68 on 11-07), the three TTS containers added to the inventory, **no removals**.
|
||||
**Updated:** [[current-state]] and [[index]] — dated change log.
|
||||
**Ledger:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit, started 23:59:37); deliveries **4 / 4 `delivered`**; **`cron_incidents` 0 → 1 unclosed** (the monitor row above, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 jobs (26 enabled / 3 disabled)** — **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13.
|
||||
**Not changed:** `errors.log` for 11-14 carries only the benign recurring `Browser Use viewing unavailable` plugin warning (~2/hour) plus the already-recorded `pre_tool_call timed out after 120s` at 00:05:40 — see [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] (no new occurrence in the 11-14 body).
|
||||
|
||||
## 2026-11-15 00:5x — corrections pass on the 2026-11-14 entry (same audit window)
|
||||
|
||||
A second read of the live DBs tightened three claims made in the entry above. All three are applied to the pages; this entry records the corrections rather than leaving two versions in the log.
|
||||
|
||||
1. **WITHDRAWN — "one net-added CT200 container unidentified."** The `61 → 65` comparison mixed two different counts: the 2026-11-07 row's **61 was an explicit *non-seeder*** figure while today's **65 was a raw `running`** figure that **includes** the transient `worldmonitor-seeder-scheduled`. A set-wise name diff settles it: live = documented **+ exactly 3** (`phonon-demo`, `ptts-openai-server`, `ptts-ws-server`), **zero removals**, and **64 non-seeder running** against 61 on 11-07. **No unexplained container exists.** [[infrastructure/docker-services]] corrected.
|
||||
2. **`cron_incidents` went 0 → 1 unclosed** — not "0 unclosed before the 00:03 alert". **138 rows** total; the open one is `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5, `failure_type: auth` — mislabelled). The monitor's **own** job is also the fleet's **single non-clean `last_status`** (`error`, `failure_streak` 1) — the **first red fleet row since 11-13**. [[systems/scheduled-tasks]], [[current-state]] and [[index]] corrected.
|
||||
3. **RESOLVED (was listed as unverified) — did the 00:03 escalation actually deliver?** Yes. `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — **3 s after** the incident row was raised, **5 s before** `alerted_at`. The **notification path worked; the detection path did not.**
|
||||
|
||||
Ledger precision as read at 00:5x: **2026-11-14: 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, 23:59:37); **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (the monitor); deliveries 11-14 **4 / 4 `delivered`** (07:06 / 07:30 / 08:32 / 12:05). Pages also updated in this pass: [[incidents/2026-11-14-mnemosyne-capture-stall]] (delivery question closed, fleet line corrected), [[systems/mnemosyne]] (new ⚠ OPEN block at the head of *Known Issues*, frontmatter to 2026-11-15, and the naive-UTC trap note extended with the monitor's own raw-UTC alert text as a second occurrence of the same trap), [[systems/scheduled-tasks]] (new `### Open 2026-11-15` block under *Known Issues* + the monitor row in the disabled table annotated — **the threshold tests the *sign* of the capture delay, not its magnitude, which is why 12 h of a growing hole raised nothing**).
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
@@ -50,6 +50,8 @@ Primary agent memory store for Hermes.
|
||||
|
||||
## Known Issues
|
||||
|
||||
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||
|
||||
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
||||
|
||||
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
|
||||
@@ -62,7 +64,7 @@ Primary agent memory store for Hermes.
|
||||
|
||||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||||
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **2026-11-14/15 recurrence, new flavour:** the monitor's own escalation text printed `last capture 2026-11-14 01:32:59` (raw UTC) for a write that landed at **09:32:59 AWST**, so the alert *reads* as a ~23 h hole when the true gap was **14 h 31 m** — the same trap, now inside the monitor's output rather than the audit's reading → [[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: Phonon TTS on CT200 (ptts-openai-server, ptts-ws-server, phonon-demo)
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-12-15
|
||||
confidence: high
|
||||
tags: [tts, phonon, ptts, docker, ct200, voice, huggingface, websocket, openai-api, kokoro, edge-tts]
|
||||
sources: [live-ct200-docker, live-ct200-docker-inspect, live-ct200-files, live-http-probe, live-state-db, live-agent-log, live-mnemosyne-db]
|
||||
---
|
||||
|
||||
# Phonon TTS on CT200
|
||||
|
||||
A **third TTS stack** arrived on CT200 on **2026-11-14** — three containers that together form a
|
||||
self-hosted "ptts" (Phonon TTS) service: an **OpenAI-compatible HTTP API** (`:8881`), a **static
|
||||
WebAssembly demo UI** (`:8882`), and a **WebSocket TTS server** (`:8883`). It runs alongside the two
|
||||
voice services that host already carried (`kokoro-tts` `:8880`, `edge-tts` `:5051`).
|
||||
|
||||
Page created by the **2026-11-14 nightly audit** from live probes (see *Verification evidence*).
|
||||
Everything below is a live read unless it says otherwise.
|
||||
|
||||
## Containers (live `docker ps` / `docker inspect`, 2026-11-15 ~00:3x AWST)
|
||||
|
||||
| Container | Image | Ports (host → container) | Created (UTC) | Last started (UTC) | Restart policy |
|
||||
|---|---|---|---|---|---|
|
||||
| `ptts-openai-server` | `xn-ptts-openai-server:v0.7.1` | `8881 → 10200` (**Wyoming also exposed, unmapped**) | `2026-11-14T02:31:56` = **10:31:56 AWST** | `2026-11-14T09:54:11` = **17:54:11 AWST** | `unless-stopped` |
|
||||
| `phonon-demo` | `nginx:alpine` | `8882 → 80` | `2026-11-14T03:00:11` = **11:00:11 AWST** | `2026-11-14T03:00:11` | `unless-stopped` |
|
||||
| `ptts-ws-server` | `ptts-ws-server:0.7.1` (**locally built**, 116 MB) | `8883 → 8080` | `2026-11-14T03:43:59` = **11:43:59 AWST** | `2026-11-14T09:54:21` = **17:54:21 AWST** | `unless-stopped` |
|
||||
|
||||
Image names/tags for the first row are as reported by `docker ps`; the ws server's image is a
|
||||
**local build** (`docker image ls ptts-ws-server` → `0.7.1`, no registry prefix), so it is not
|
||||
re-pullable — **the build inputs are not in the wiki and were not located in this audit.**
|
||||
|
||||
## Endpoints (verified live from CT460 over the tailnet, 2026-11-15)
|
||||
|
||||
| URL | What answers |
|
||||
|---|---|
|
||||
| `http://<CT200 tailnet IP>:8881/health` | health JSON, HTTP 200 |
|
||||
| `http://<CT200 tailnet IP>:8881/v1/models` | OpenAI-shaped model list → **`gradium/phonon-40m-en`** |
|
||||
| `http://<CT200 tailnet IP>:8881/v1/audio/speech` | **POST works** — returned a valid MP3 (test voice `nova`) |
|
||||
| `http://<CT200 tailnet IP>:8882/` | the demo UI — `nginx:alpine`, **19,212 B** body, byte-identical in size to the file below |
|
||||
| `http://<CT200 tailnet IP>:8883/` | **HTTP 404 on a plain GET** — expected: it is a WebSocket endpoint (`/speech/tts`) |
|
||||
|
||||
The `:8881` API is the piece that other services can consume (OpenAI-compatible `/v1/audio/speech`).
|
||||
|
||||
## Files on CT200 (owner/paths verified by live `ls -la`)
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `/opt/ptts-openai-server/` | `run.sh` (861 B, `2026-11-14 09:01` local) + `.env` (**47 B, mode 600**, `2026-11-14 09:00` local — holds the HuggingFace token for the gated model; **value not recorded here**) |
|
||||
| `/opt/phonon-demo/index.html` | the demo page, `2026-11-14 12:30` local; title *"Phonon TTS - WebAssembly Demo"*; talks to **`:8883`** only (no `8881` reference) |
|
||||
| `/opt/phonon-demo/phonon-tts/` | the WASM/JS assets (`device.js`, `fetch.js`, `hub.js`, `gpu/`, `index.js`, `index.d.ts`, `LICENSE-APACHE`) — i.e. the upstream `phonon-demo` front-end |
|
||||
| `/opt/ptts-voices/` | **7 entries** (`2026-11-14 17:54` local): six `.safetensors` voices — `anthony.safetensors`, `anthony-clone.safetensors`, `anthony-m.safetensors`, `anthony-november.safetensors`, `elizabeth.safetensors`, `grace.safetensors` — plus a `voices.jsonl` manifest |
|
||||
|
||||
The ws server logged **`num_voices=6` at 2026-11-14 04:42:52 UTC (12:42:52 AWST)** and
|
||||
**`num_voices=7` at 09:54:24 UTC (17:54:24 AWST)** — so the voice set was still being changed that
|
||||
evening. **Provenance of the voice files is not established**: the naming suggests cloned/derived
|
||||
voices, but nothing in the retained logs confirms whose they are or how they were produced.
|
||||
|
||||
## Deployment timeline (2026-11-14)
|
||||
|
||||
| AWST | Event | Source |
|
||||
|---|---|---|
|
||||
| 08:35 – 08:45 | First Phonon install session: `ptts-openai-server` brought up on `:8881`; the HuggingFace token returned **401** on the gated model and was replaced | session transcript (`state.db`) |
|
||||
| 08:52 – 09:03 | Second session: model switched to `gradium/phonon-40m-en`; `/opt/ptts-openai-server/run.sh` (861 B) and its `600` `.env` written at **09:00–09:01** | session transcript + live `ls` |
|
||||
| 10:31:56 | `ptts-openai-server` **re-created** (2nd recreation; the 09:01 container was not the live one) | live `docker inspect` |
|
||||
| 11:00:11 | `phonon-demo` (`nginx:alpine`) created → demo UI on `:8882` | live `docker inspect` |
|
||||
| 11:43:59 | `ptts-ws-server:0.7.1` created → WebSocket TTS on `:8883` | live `docker inspect` |
|
||||
| 12:30 | `/opt/phonon-demo/index.html` last written | live `ls` mtime |
|
||||
| 12:42:52 | ws server logs `num_voices=6` | live `docker logs` |
|
||||
| 17:54:11 / 17:54:21 | **both** ptts servers restarted together; `/opt/ptts-voices` written; ws logs `num_voices=7` | live `docker inspect` + `ls` + `logs` |
|
||||
|
||||
⚠ **Provenance gap.** The two retained interactive sessions in `state.db` cover only **08:35–09:03
|
||||
AWST**. Every change from **10:31 onward** — the demo UI, the ws-server build, the voice files, the
|
||||
17:54 restart — is **not attributable to any retained transcript**: the only sessions active in
|
||||
`state.db` between 09:03 and 23:59 on 11-14 are cron runs and a heartbeat "Pulse" session (495
|
||||
messages, none containing docker/TTS content). **Working hypothesis, not proven: the later work was
|
||||
done by another estate agent or a session not retained in this profile's store.** Do not treat the
|
||||
08:35/08:52 sessions as the whole of this deployment.
|
||||
|
||||
## Why it is on `8881` and not `8880`
|
||||
|
||||
`8880` was already taken by **`kokoro-tts`** on the same host (`ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1`,
|
||||
declared in the ReadLater compose file → [[infrastructure/docker-services]]), and `edge-tts` sits on
|
||||
`:5051`. Phonon is therefore the **third** voice backend on CT200 — three TTS stacks on one LXC.
|
||||
|
||||
## Operations
|
||||
|
||||
```bash
|
||||
# reach CT200 (the PVE node is 100.96.100.82; the wiki's CT200 row carries the tailnet IP)
|
||||
ssh root@100.96.100.82 'pct exec 200 -- docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -Ei "ptts|phonon"'
|
||||
# logs (the ws server is the talkative one)
|
||||
ssh root@100.96.100.82 'pct exec 200 -- docker logs --tail 40 ptts-ws-server'
|
||||
# smoke test the OpenAI-compatible path from another host on the tailnet
|
||||
curl -s -X POST "http://<CT200 tailnet IP>:8881/v1/audio/speech" -H 'Content-Type: application/json' \
|
||||
-d '{"model":"gradium/phonon-40m-en","input":"test","voice":"nova"}' -o /tmp/probe.mp3 && file /tmp/probe.mp3
|
||||
```
|
||||
|
||||
## Open items / watch
|
||||
|
||||
1. **The HuggingFace token is a standing single point of failure.** Both `ptts-openai-server` and
|
||||
`ptts-ws-server` serve a **gated** model, and the token has already expired once (2026-11-14
|
||||
08:35–08:45 → replaced). The token lives in `~/.hermes/.env` on CT460 and in
|
||||
`/opt/ptts-openai-server/.env` (mode 600) on CT200; **whether the ws container reads the same file
|
||||
was not confirmed in this audit.** A silent expiry would take the whole stack down again.
|
||||
2. **`ptts-ws-server:0.7.1` is a local-only image** — no build source recorded here.
|
||||
3. **Attribution gap** for everything after 09:03 AWST (see the timeline note).
|
||||
4. **Nothing in the estate consumes `:8881` yet**, as far as this audit could see: it was verified by
|
||||
probe, not by a caller. If it is wired into Hermes TTS later, the provider config belongs in
|
||||
[[systems/hermes-agent]] / the TTS configuration, not here.
|
||||
5. The upstream project's own naming is inconsistent (`ptts-*` containers, `phonon-*` model/page, a
|
||||
"gradium" publisher prefix) — the same stack appears under three names.
|
||||
|
||||
## Related
|
||||
|
||||
- [[infrastructure/docker-services]] — CT200 inventory and container counts (65 running / 72 total as of 2026-11-15)
|
||||
- [[systems/mnemosyne]] — the capture stall recorded the same night ([[incidents/2026-11-14-mnemosyne-capture-stall]])
|
||||
- [[current-state]] — dated change log
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||
retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` /
|
||||
1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173,
|
||||
11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32,
|
||||
12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status:
|
||||
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
||||
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
||||
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
||||
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
|
||||
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
||||
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
|
||||
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
|
||||
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
|
||||
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
||||
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
||||
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
||||
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
|
||||
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
|
||||
@@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
||||
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Open 2026-11-15
|
||||
|
||||
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
||||
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
|
||||
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
|
||||
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
|
||||
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
|
||||
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
|
||||
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
|
||||
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
|
||||
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
|
||||
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
||||
misreading trap.
|
||||
|
||||
### Resolved 2026-11-02
|
||||
|
||||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||||
|
||||
Reference in New Issue
Block a user