audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections

omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
This commit is contained in:
hermes
2026-10-16 00:19:14 +08:00
parent deb6a8eec2
commit 2061b8607b
7 changed files with 328 additions and 11 deletions
+12 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
updated: 2026-10-16
verified_on: 2026-10-16
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -94,6 +94,16 @@ Primary agent memory store for Hermes.
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
- Mnemosyne Embedding Coverage every 6h — job `b63ed5817212`, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
- ⚠ **Blast-radius note (2026-10-16).** Both monitors above (`ea79a2107e36` Health every 12 h,
`b63ed5817212` Embedding Coverage every 6 h) are pinned to **`custom:omni-101`** — the same node the
memory system's own reporting depends on. During the **2026-10-14 → 10-15 `omniroute-1` outage both
failed on every slot**, so **no coverage figures were refreshed for ~34 hours**; the last explicit read
remains the **2026-10-12 12:02** run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the
**2026-10-16 00:09:48** Health Monitor run is the first clean read since. The two script-only jobs
(`consolidate_mnemosyne.sh`, `mnemosyne-backup.sh`) were **unaffected** — they make no model call.
Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that
cannot fire during a router outage is exactly the case this has to cover.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
## History and Evidence
- **2026-10-03:** vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].