audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07 (httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error). 36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1). STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence. New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live). Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute, systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
+12
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -94,6 +94,16 @@ Primary agent memory store for Hermes.
|
||||
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
||||
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
|
||||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
|
||||
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
|
||||
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
|
||||
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
|
||||
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
|
||||
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
|
||||
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
|
||||
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
|
||||
cannot fire during a router outage is exactly the case this has to cover.
|
||||
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
|
||||
## History and Evidence
|
||||
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
|
||||
Reference in New Issue
Block a user