wiki: 2026-11-14 nightly audit + corrections — Phonon TTS on CT200, Mnemosyne capture stall open
- new: systems/phonon-tts.md, incidents/2026-11-14-mnemosyne-capture-stall.md - CT200 inventory re-verified: 64 non-seeder running / 72 total (delta = the TTS trio, no removals) - fleet ledger: 11-14 = 167 rows / 166 completed / 0 failed; first red last_status since 11-13 (Mnemosyne Health Monitor, error, streak 1); cron_incidents 0 -> 1 unclosed (138 rows) - corrections pass: withdrew the 'unidentified extra container' claim; closed the alert-delivery question (delivered 00:03:39, for_failure=1) - updated: index.md, current-state.md, log.md, systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/docker-services.md
This commit is contained in:
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-14
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,23 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||
retired, disabled or repinned). Ledger — **2026-11-14: 167 rows, 166 `completed` / 0 `failed` /
|
||||
1 `running`** (this audit, `cbfe434a3b5e`, started 23:59:37). Trend: **11-12 176/176, 11-13 174/173,
|
||||
11-14 167/166**. **Deliveries 11-14: 4 rows, 4 / 4 `delivered`, 0 `for_failure`** (07:06, 07:30, 08:32,
|
||||
12:05). ⚠ **Non-clean `last_status` 0 -> 1**: **`ea79a2107e36` (Mnemosyne Health Monitor)**, `last_status:
|
||||
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
||||
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
||||
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
||||
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
|
||||
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
||||
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
|
||||
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
|
||||
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
|
||||
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
||||
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
||||
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
||||
consecutive zero-`failed` day — and the fleet went fully clean.** Fleet **29 (26 enabled / 3 disabled)**,
|
||||
an **eighth consecutive window at the same count** (nothing added, retired, disabled or repinned).
|
||||
@@ -287,10 +304,25 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
||||
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. (⚠ **2026-11-15:** the job is enabled and now carries `last_status: error` / `failure_streak: 1` from its 00:03:44 run — [[incidents/2026-11-14-mnemosyne-capture-stall]].) The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Open 2026-11-15
|
||||
|
||||
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
||||
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
|
||||
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
|
||||
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
|
||||
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
|
||||
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
|
||||
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
|
||||
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
|
||||
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
|
||||
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
||||
misreading trap.
|
||||
|
||||
### Resolved 2026-11-02
|
||||
|
||||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||||
|
||||
Reference in New Issue
Block a user