wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
This commit is contained in:
@@ -2605,3 +2605,23 @@ hook timeout came back and reopened its incident.**
|
||||
2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes
|
||||
(2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter),
|
||||
`log.md` (this entry). **No page deleted, no decision reversed, no secret written.**
|
||||
|
||||
## 2026-11-14 (nightly audit; window 2026-11-13 23:00 → 2026-11-14 23:59 AWST)
|
||||
|
||||
**New:** [[systems/phonon-tts]] — the third TTS stack on CT200: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`). Verified live from CT460 — `/health` 200, `/v1/models`, and a POST to `/v1/audio/speech` returning a valid MP3; `:8883` 404s on a plain GET because it is a WebSocket endpoint. `/opt/ptts-voices` holds 7 voices. Page carries the container/timeline table and flags the **provenance gap after 09:03 AWST** — no retained session in this profile covers the 10:31 / 11:00 / 11:43 / 17:54 changes.
|
||||
**New:** [[incidents/2026-11-14-mnemosyne-capture-stall]] — **open incident**: Mnemosyne has captured nothing since **2026-11-14 09:32:59 AWST** (`working_memory` and `episodic_memory` stop in the same minute; last 8 working writes all 01:01–01:32 UTC). The health monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state `alerted`, unclosed) after ~12 h of lead time in which the same hole was logged **ℹ️ informational** at 12:00 AWST (`capture delay -589.5 min`). ~15 h of desktop/Telegram/cron context uncaptured; the 23:30 sleep cycle consolidated **0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime 2026-11-13 18:03; YAML walk clean). Root cause **not established**.
|
||||
**Updated:** [[infrastructure/docker-services]] — CT200 re-verified 2026-11-15: **65 running / 72 total** (was 61 / 68 on 11-07), the three TTS containers added to the inventory, **no removals**.
|
||||
**Updated:** [[current-state]] and [[index]] — dated change log.
|
||||
**Ledger:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit, started 23:59:37); deliveries **4 / 4 `delivered`**; **`cron_incidents` 0 → 1 unclosed** (the monitor row above, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 jobs (26 enabled / 3 disabled)** — **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13.
|
||||
**Not changed:** `errors.log` for 11-14 carries only the benign recurring `Browser Use viewing unavailable` plugin warning (~2/hour) plus the already-recorded `pre_tool_call timed out after 120s` at 00:05:40 — see [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] (no new occurrence in the 11-14 body).
|
||||
|
||||
## 2026-11-15 00:5x — corrections pass on the 2026-11-14 entry (same audit window)
|
||||
|
||||
A second read of the live DBs tightened three claims made in the entry above. All three are applied to the pages; this entry records the corrections rather than leaving two versions in the log.
|
||||
|
||||
1. **WITHDRAWN — "one net-added CT200 container unidentified."** The `61 → 65` comparison mixed two different counts: the 2026-11-07 row's **61 was an explicit *non-seeder*** figure while today's **65 was a raw `running`** figure that **includes** the transient `worldmonitor-seeder-scheduled`. A set-wise name diff settles it: live = documented **+ exactly 3** (`phonon-demo`, `ptts-openai-server`, `ptts-ws-server`), **zero removals**, and **64 non-seeder running** against 61 on 11-07. **No unexplained container exists.** [[infrastructure/docker-services]] corrected.
|
||||
2. **`cron_incidents` went 0 → 1 unclosed** — not "0 unclosed before the 00:03 alert". **138 rows** total; the open one is `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5, `failure_type: auth` — mislabelled). The monitor's **own** job is also the fleet's **single non-clean `last_status`** (`error`, `failure_streak` 1) — the **first red fleet row since 11-13**. [[systems/scheduled-tasks]], [[current-state]] and [[index]] corrected.
|
||||
3. **RESOLVED (was listed as unverified) — did the 00:03 escalation actually deliver?** Yes. `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — **3 s after** the incident row was raised, **5 s before** `alerted_at`. The **notification path worked; the detection path did not.**
|
||||
|
||||
Ledger precision as read at 00:5x: **2026-11-14: 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, 23:59:37); **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (the monitor); deliveries 11-14 **4 / 4 `delivered`** (07:06 / 07:30 / 08:32 / 12:05). Pages also updated in this pass: [[incidents/2026-11-14-mnemosyne-capture-stall]] (delivery question closed, fleet line corrected), [[systems/mnemosyne]] (new ⚠ OPEN block at the head of *Known Issues*, frontmatter to 2026-11-15, and the naive-UTC trap note extended with the monitor's own raw-UTC alert text as a second occurrence of the same trap), [[systems/scheduled-tasks]] (new `### Open 2026-11-15` block under *Known Issues* + the monitor row in the disabled table annotated — **the threshold tests the *sign* of the capture delay, not its magnitude, which is why 12 h of a growing hole raised nothing**).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user