omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07 (httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error). 36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1). STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence. New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live). Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute, systems/mnemosyne (monitor blast-radius note), current-state, index, log.
13 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne | system | active | 2026-07-22 | 2026-10-16 | 2026-10-16 | high |
|
|
Mnemosyne
Purpose
Primary agent memory store for Hermes.
Current Configuration
- Provider:
mnemosyne - Enabled: true
- Auto sleep: true
- Default scope:
session - Memory char limit: 5000
- User char limit: 3000
- Vector type: int8
- Profile isolation: true
- Shared surface read: true
- Shared surface path:
data/shared/mnemosyne.db - Skip contexts: cron, flush, subagent, background, skill_loop
- Reflect max calls per session: 3
- Sync roles: user
- Flush every: 6 turns
- Embedding model:
BAAI/bge-small-en-v1.5(384 dims) - Embeddings via API: false (local fastembed)
Database Stats (live, 2026-10-04)
| Table | Rows | Notes |
|---|---|---|
working_memory |
773 | Active memories |
episodic_memory |
1,047 | Consolidated summaries |
memory_embeddings |
760 | Working-tier embeddings |
memory (legacy) |
2,181 | Old table, not migrated |
vec_working / vec_working_rowids |
760 | Working vector index |
vec_episodes / vec_episodes_rowids |
1,035 | Episodic vector index |
vec_facts_rowids |
0 | memoria_facts is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional |
Embedding coverage (live, 2026-10-04): 1,759/2,194 memories embedded (80.2%) — working 726/752 (96.5%), episodic 1,033/1,039 (99.4%); 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (b63ed5817212, every 6h, Telegram topic 102).
Known Issues
Tier counts (live, 2026-10-09 22:27): working 773, episodic 1,047, canonical_facts 97, facts 871, memoria_facts 407, memoria_preferences 132, conflicts 152 (1 open — id 152, contradiction, created 2026-10-09 13:11:38), working embeddings 760/773, episodic embeddings 1,043/1,047 (99.6%). Ranking relevance is the live issue, not coverage — see incidents/2026-10-09-memory-recall-relevance.
Tier counts (live, 2026-10-12 12:02 — health job ea79a2107e36, green): working 760/786 embedded (96.7%), episodic 1050/1056 (99.4%) — both above the 95% floor, no reindex needed (b63ed5817212 continues to backfill). Orphan embeddings 0. FTS5 exactly aligned: fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876. 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is not the live issue — ranking relevance is (see incidents/2026-10-09-memory-recall-relevance).
Tier counts (live, 2026-10-12 00:1x): working 786, episodic 1,056, facts 876 (fts_facts 876 — FTS aligned), fts_working 786, memoria_facts 410, conflicts 152. Embeddings: working 773/786 (98.3%), episodic 1,052/1,056 (99.6%) — both above the 95% reindex threshold; the coverage job b63ed5817212 backfilled 15 vectors since the 2026-10-11 00:08 check.
Capture clock is naive UTC — never compare it to AWST now (2026-10-11 false alarm). At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing working_memory MAX(created_at) = 2026-10-10 13:24:53 against the local clock. created_at is stored as a naive UTC string (the same trap as consolidation_log), so the real gap was 2.7 h: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — working_memory gained 2 rows on 2026-10-11 (11:19:37 and 11:51:39 UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- Resolved by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is 96.5% working / 99.4% episodic, and new embeddings are still being generated. The figures above are kept as the before-state only.
- Standard repair if coverage drops below 95% working:
~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes(the Embedding Coverage jobb63ed5817212triggers the same path every 6h).
Vector recall silently ran FTS-only (2026-10-03 — repaired)
fastembedwas absent from every uv-managed runtime, so the dense leg ofmnemosyne_recallproduced no candidates and recall fell back to FTS5/keyword scoring without raising an error. Sessions kept working, which is why it went unnoticed.- Compounded by
FASTEMBED_CACHE_PATHbeing unset in~/.hermes/.env, and by the embedding worker sometimes launching in isolated (python -I) mode, which does not inheritPYTHONPATH. - Repaired 2026-10-03:
fastembedinstalled into the bootstrap-selected runtime,FASTEMBED_CACHE_PATHpinned, andhermes-runtime-deps-guard.shextended to assert fastembed + cache path + model dir every 15 min alongside its existingdotenv/mnemosynechecks. - CLI caveat — do not misread
mnemosyne doctor. Doctor and the recall CLI run under the legacyvenv/(Python 3.11), where thesqlite_vecextension is not importable; doctor therefore printsvec0: present_but_unloadableand marks the vector tiersunavailableeven while the gateway embeds normally. Judge vector health with~/.hermes/scripts/check_embedding_coverage.py, not with doctor'svec0line. - Benign noise:
pthread_setaffinity_np failed … mask: {0,}from onnxruntime inmcp-stderr.logis a container affinity warning emitted on every model load — not an error. - See incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
Profile isolation chosen over shared surface
|- The ops profile was enabled with mnemosyne on 2026-08-02
|- Decision: profile_isolation: true was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|- data/shared/mnemosyne.db exists with 0 rows; cross-profile sharing is intentionally unused
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|- Config in ~/.hermes/profiles/ops/config.yaml: memory.mnemosyne.profile_isolation: true, allow_tool_override: false, shared_surface_read: false
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
Legacy table not migrated
- 2,181 rows in the old
memorytable are not in the newworking_memory/episodic_memorysystem - These are only accessible via the legacy
memorytool, not viamnemosyne_recall
Gateway notifications were captured as user memories (fixed 2026-09-30)
- Hermes's background-process notifications (long-running
terminaljobs reporting completion) were being written into Mnemosyne as[USER] [Anthony Martin]memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words. - 36 such rows were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
- Two canonical facts were also wrong because of the same class of drift:
hermes_cron_model_defaultnamed the removed providercustom:omniinstead ofcustom:omni-101, and the Daily Marketing Job Radar's model fact was stale. Both corrected. - Rule: treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
Known Management Jobs
- Mnemosyne Auto-Consolidate every 30m — script:
consolidate_mnemosyne.sh, mode: no-agent - Mnemosyne DB Backup weekly Sunday 03:00 — script:
mnemosyne-backup.sh, mode: no-agent - Mnemosyne Health Monitor every 12h — skill:
mnemosyne-operations, reports only on exception - Mnemosyne Embedding Coverage every 6h — job
b63ed5817212, alerts to Telegram topic 102 only when working-tier coverage drops below 95% - Mnemosyne Watchdog every 15m — script:
mnemosyne-watchdog.py, mode: no-agent - ⚠ Blast-radius note (2026-10-16). Both monitors above (
ea79a2107e36Health every 12 h,b63ed5817212Embedding Coverage every 6 h) are pinned tocustom:omni-101— the same node the memory system's own reporting depends on. During the 2026-10-14 → 10-15omniroute-1outage both failed on every slot, so no coverage figures were refreshed for ~34 hours; the last explicit read remains the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the 2026-10-16 00:09:48 Health Monitor run is the first clean read since. The two script-only jobs (consolidate_mnemosyne.sh,mnemosyne-backup.sh) were unaffected — they make no model call. Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that cannot fire during a router outage is exactly the case this has to cover. → incidents/2026-10-14-omniroute-node-connection-refused-storm
History and Evidence
- 2026-10-03: vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
- 2026-09-30 memory-hygiene pass: 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (
hermes_cron_model_default→custom:omni-101, job-radar model). No unresolved conflicts left in the working tier. - 2026-08-19 Mnemosyne Watchdog incident: Watchdog job (
96eb95ce12f2) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See systems/scheduled-tasks Known Issues section. - 2026-08-18 incident: Working memory capture (
sync_turn) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions inrun_agent.py:_sync_external_memory_for_turn(). Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See incidents/2026-08-18-mnemosyne-capture-pipeline. - 2026-08-01 incident: Automatic conversational capture was broken by stringified structured YAML, especially
sync_roles: "['user']". Fixed by rewriting with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure. - 2026-08-01 cleanup: Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
- 2026-07-29 incident:
memory.mnemosyne.tools: Nonecaused schema loading warnings. Fixed by removing the line. See incidents/2026-07-29-mnemosyne-relay. - 2026-07-29 relay scope corruption:
RuntimeError: scope handle is not at the top of the stack— 3 occurrences, self-resolved.
Related
- concepts/agent-memory
- comparisons/memory-backends
- incidents/2026-08-01-mnemosyne-capture-failure
- incidents/2026-08-18-mnemosyne-capture-pipeline
- incidents/2026-07-29-mnemosyne-relay
Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)
- Gateway load path repaired. The gateway's
activate_dependencies()selects one committed uv env (795020dbf24343cd9c52df24723dca83); it was missingmnemosyne/mnemosyne_hermesand the native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19 (registered (40 tools)->activated). Detail: incidents/2026-09-28-hermes-managed-runtime-deps. - "Capture silent since 10-09 15:20" was a false alarm. It resolved to no interactive sessions
in the window - the window was cron-only traffic. What the surrounding hours did capture was
dominated by gateway background-process notifications stored as
[USER] [Anthony Martin] [IMPORTANT: Background process ...]rows: recall-visible, low signal. A capture filter would help; no fix applied (carried open item). - Coverage script executable again:
~/.hermes/scripts/check_embedding_coverage.pynow carries mode0755, so the monitor's non-executable finding is resolved. - Open, unverified: 13 orphan rows in
memory_embeddings(oldest 2026-08-22) reported by a 2026-10-10 monitor run - neither confirmed nor cleaned by this audit.