wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
This commit is contained in:
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-13
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
@@ -50,6 +50,8 @@ Primary agent memory store for Hermes.
|
||||
|
||||
## Known Issues
|
||||
|
||||
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||
|
||||
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
|
||||
|
||||
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
|
||||
@@ -62,7 +64,7 @@ Primary agent memory store for Hermes.
|
||||
|
||||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||||
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. **2026-11-14/15 recurrence, new flavour:** the monitor's own escalation text printed `last capture 2026-11-14 01:32:59` (raw UTC) for a write that landed at **09:32:59 AWST**, so the alert *reads* as a ~23 h hole when the true gap was **14 h 31 m** — the same trap, now inside the monitor's output rather than the audit's reading → [[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: Phonon TTS on CT200 (ptts-openai-server, ptts-ws-server, phonon-demo)
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-12-15
|
||||
confidence: high
|
||||
tags: [tts, phonon, ptts, docker, ct200, voice, huggingface, websocket, openai-api, kokoro, edge-tts]
|
||||
sources: [live-ct200-docker, live-ct200-docker-inspect, live-ct200-files, live-http-probe, live-state-db, live-agent-log, live-mnemosyne-db]
|
||||
---
|
||||
|
||||
# Phonon TTS on CT200
|
||||
|
||||
A **third TTS stack** arrived on CT200 on **2026-11-14** — three containers that together form a
|
||||
self-hosted "ptts" (Phonon TTS) service: an **OpenAI-compatible HTTP API** (`:8881`), a **static
|
||||
WebAssembly demo UI** (`:8882`), and a **WebSocket TTS server** (`:8883`). It runs alongside the two
|
||||
voice services that host already carried (`kokoro-tts` `:8880`, `edge-tts` `:5051`).
|
||||
|
||||
Page created by the **2026-11-14 nightly audit** from live probes (see *Verification evidence*).
|
||||
Everything below is a live read unless it says otherwise.
|
||||
|
||||
## Containers (live `docker ps` / `docker inspect`, 2026-11-15 ~00:3x AWST)
|
||||
|
||||
| Container | Image | Ports (host → container) | Created (UTC) | Last started (UTC) | Restart policy |
|
||||
|---|---|---|---|---|---|
|
||||
| `ptts-openai-server` | `xn-ptts-openai-server:v0.7.1` | `8881 → 10200` (**Wyoming also exposed, unmapped**) | `2026-11-14T02:31:56` = **10:31:56 AWST** | `2026-11-14T09:54:11` = **17:54:11 AWST** | `unless-stopped` |
|
||||
| `phonon-demo` | `nginx:alpine` | `8882 → 80` | `2026-11-14T03:00:11` = **11:00:11 AWST** | `2026-11-14T03:00:11` | `unless-stopped` |
|
||||
| `ptts-ws-server` | `ptts-ws-server:0.7.1` (**locally built**, 116 MB) | `8883 → 8080` | `2026-11-14T03:43:59` = **11:43:59 AWST** | `2026-11-14T09:54:21` = **17:54:21 AWST** | `unless-stopped` |
|
||||
|
||||
Image names/tags for the first row are as reported by `docker ps`; the ws server's image is a
|
||||
**local build** (`docker image ls ptts-ws-server` → `0.7.1`, no registry prefix), so it is not
|
||||
re-pullable — **the build inputs are not in the wiki and were not located in this audit.**
|
||||
|
||||
## Endpoints (verified live from CT460 over the tailnet, 2026-11-15)
|
||||
|
||||
| URL | What answers |
|
||||
|---|---|
|
||||
| `http://<CT200 tailnet IP>:8881/health` | health JSON, HTTP 200 |
|
||||
| `http://<CT200 tailnet IP>:8881/v1/models` | OpenAI-shaped model list → **`gradium/phonon-40m-en`** |
|
||||
| `http://<CT200 tailnet IP>:8881/v1/audio/speech` | **POST works** — returned a valid MP3 (test voice `nova`) |
|
||||
| `http://<CT200 tailnet IP>:8882/` | the demo UI — `nginx:alpine`, **19,212 B** body, byte-identical in size to the file below |
|
||||
| `http://<CT200 tailnet IP>:8883/` | **HTTP 404 on a plain GET** — expected: it is a WebSocket endpoint (`/speech/tts`) |
|
||||
|
||||
The `:8881` API is the piece that other services can consume (OpenAI-compatible `/v1/audio/speech`).
|
||||
|
||||
## Files on CT200 (owner/paths verified by live `ls -la`)
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `/opt/ptts-openai-server/` | `run.sh` (861 B, `2026-11-14 09:01` local) + `.env` (**47 B, mode 600**, `2026-11-14 09:00` local — holds the HuggingFace token for the gated model; **value not recorded here**) |
|
||||
| `/opt/phonon-demo/index.html` | the demo page, `2026-11-14 12:30` local; title *"Phonon TTS - WebAssembly Demo"*; talks to **`:8883`** only (no `8881` reference) |
|
||||
| `/opt/phonon-demo/phonon-tts/` | the WASM/JS assets (`device.js`, `fetch.js`, `hub.js`, `gpu/`, `index.js`, `index.d.ts`, `LICENSE-APACHE`) — i.e. the upstream `phonon-demo` front-end |
|
||||
| `/opt/ptts-voices/` | **7 entries** (`2026-11-14 17:54` local): six `.safetensors` voices — `anthony.safetensors`, `anthony-clone.safetensors`, `anthony-m.safetensors`, `anthony-november.safetensors`, `elizabeth.safetensors`, `grace.safetensors` — plus a `voices.jsonl` manifest |
|
||||
|
||||
The ws server logged **`num_voices=6` at 2026-11-14 04:42:52 UTC (12:42:52 AWST)** and
|
||||
**`num_voices=7` at 09:54:24 UTC (17:54:24 AWST)** — so the voice set was still being changed that
|
||||
evening. **Provenance of the voice files is not established**: the naming suggests cloned/derived
|
||||
voices, but nothing in the retained logs confirms whose they are or how they were produced.
|
||||
|
||||
## Deployment timeline (2026-11-14)
|
||||
|
||||
| AWST | Event | Source |
|
||||
|---|---|---|
|
||||
| 08:35 – 08:45 | First Phonon install session: `ptts-openai-server` brought up on `:8881`; the HuggingFace token returned **401** on the gated model and was replaced | session transcript (`state.db`) |
|
||||
| 08:52 – 09:03 | Second session: model switched to `gradium/phonon-40m-en`; `/opt/ptts-openai-server/run.sh` (861 B) and its `600` `.env` written at **09:00–09:01** | session transcript + live `ls` |
|
||||
| 10:31:56 | `ptts-openai-server` **re-created** (2nd recreation; the 09:01 container was not the live one) | live `docker inspect` |
|
||||
| 11:00:11 | `phonon-demo` (`nginx:alpine`) created → demo UI on `:8882` | live `docker inspect` |
|
||||
| 11:43:59 | `ptts-ws-server:0.7.1` created → WebSocket TTS on `:8883` | live `docker inspect` |
|
||||
| 12:30 | `/opt/phonon-demo/index.html` last written | live `ls` mtime |
|
||||
| 12:42:52 | ws server logs `num_voices=6` | live `docker logs` |
|
||||
| 17:54:11 / 17:54:21 | **both** ptts servers restarted together; `/opt/ptts-voices` written; ws logs `num_voices=7` | live `docker inspect` + `ls` + `logs` |
|
||||
|
||||
⚠ **Provenance gap.** The two retained interactive sessions in `state.db` cover only **08:35–09:03
|
||||
AWST**. Every change from **10:31 onward** — the demo UI, the ws-server build, the voice files, the
|
||||
17:54 restart — is **not attributable to any retained transcript**: the only sessions active in
|
||||
`state.db` between 09:03 and 23:59 on 11-14 are cron runs and a heartbeat "Pulse" session (495
|
||||
messages, none containing docker/TTS content). **Working hypothesis, not proven: the later work was
|
||||
done by another estate agent or a session not retained in this profile's store.** Do not treat the
|
||||
08:35/08:52 sessions as the whole of this deployment.
|
||||
|
||||
## Why it is on `8881` and not `8880`
|
||||
|
||||
`8880` was already taken by **`kokoro-tts`** on the same host (`ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1`,
|
||||
declared in the ReadLater compose file → [[infrastructure/docker-services]]), and `edge-tts` sits on
|
||||
`:5051`. Phonon is therefore the **third** voice backend on CT200 — three TTS stacks on one LXC.
|
||||
|
||||
## Operations
|
||||
|
||||
```bash
|
||||
# reach CT200 (the PVE node is 100.96.100.82; the wiki's CT200 row carries the tailnet IP)
|
||||
ssh root@100.96.100.82 'pct exec 200 -- docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -Ei "ptts|phonon"'
|
||||
# logs (the ws server is the talkative one)
|
||||
ssh root@100.96.100.82 'pct exec 200 -- docker logs --tail 40 ptts-ws-server'
|
||||
# smoke test the OpenAI-compatible path from another host on the tailnet
|
||||
curl -s -X POST "http://<CT200 tailnet IP>:8881/v1/audio/speech" -H 'Content-Type: application/json' \
|
||||
-d '{"model":"gradium/phonon-40m-en","input":"test","voice":"nova"}' -o /tmp/probe.mp3 && file /tmp/probe.mp3
|
||||
```
|
||||
|
||||
## Open items / watch
|
||||
|
||||
1. **The HuggingFace token is a standing single point of failure.** Both `ptts-openai-server` and
|
||||
`ptts-ws-server` serve a **gated** model, and the token has already expired once (2026-11-14
|
||||
08:35–08:45 → replaced). The token lives in `~/.hermes/.env` on CT460 and in
|
||||
`/opt/ptts-openai-server/.env` (mode 600) on CT200; **whether the ws container reads the same file
|
||||
was not confirmed in this audit.** A silent expiry would take the whole stack down again.
|
||||
2. **`ptts-ws-server:0.7.1` is a local-only image** — no build source recorded here.
|
||||
3. **Attribution gap** for everything after 09:03 AWST (see the timeline note).
|
||||
4. **Nothing in the estate consumes `:8881` yet**, as far as this audit could see: it was verified by
|
||||
probe, not by a caller. If it is wired into Hermes TTS later, the provider config belongs in
|
||||
[[systems/hermes-agent]] / the TTS configuration, not here.
|
||||
5. The upstream project's own naming is inconsistent (`ptts-*` containers, `phonon-*` model/page, a
|
||||
"gradium" publisher prefix) — the same stack appears under three names.
|
||||
|
||||
## Related
|
||||
|
||||
- [[infrastructure/docker-services]] — CT200 inventory and container counts (65 running / 72 total as of 2026-11-15)
|
||||
- [[systems/mnemosyne]] — the capture stall recorded the same night ([[incidents/2026-11-14-mnemosyne-capture-stall]])
|
||||
- [[current-state]] — dated change log
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||
retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` /
|
||||
1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173,
|
||||
11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32,
|
||||
12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status:
|
||||
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
||||
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
||||
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
||||
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
|
||||
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
||||
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
|
||||
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
|
||||
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
|
||||
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
||||
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
||||
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
||||
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
|
||||
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
|
||||
@@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
||||
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Open 2026-11-15
|
||||
|
||||
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
||||
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
|
||||
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
|
||||
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
|
||||
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
|
||||
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
|
||||
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
|
||||
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
|
||||
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
|
||||
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
||||
misreading trap.
|
||||
|
||||
### Resolved 2026-11-02
|
||||
|
||||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||||
|
||||
Reference in New Issue
Block a user