wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open

- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md
- CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals)
- fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13
  (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows)
- corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery
  question (delivered 00:03:39, for_failure=1)
- updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md,
  infrastructure/docker-services.md
This commit is contained in:
Tony0410
2026-11-15 00:42:01 +08:00
parent 16767ef562
commit 4a6e86948d
8 changed files with 311 additions and 16 deletions
+36 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-14
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-15
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` /
1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173,
11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32,
12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status:
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
@@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues
### Open 2026-11-15
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
misreading trap.
### Resolved 2026-11-02
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**