Files
agent-estate-wiki/systems/mnemosyne.md
T
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

27 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Mnemosyne system active 2026-07-22 2026-11-20 2026-11-20 2026-11-21 high
memory
mnemosyne
backend
live-probe
mnemosyne-db
config
ops-profile-config
live-secrets-scan
live-state-db
live-health-monitor-log

Mnemosyne

Purpose

Primary agent memory store for Hermes.

Current Configuration

  • Provider: mnemosyne
  • Enabled: true
  • Auto sleep: true
  • Default scope: session
  • Memory char limit: 5000
  • User char limit: 3000
  • Vector type: int8
  • Profile isolation: true
  • Shared surface read: true
  • Shared surface path: data/shared/mnemosyne.db
  • Skip contexts: cron, flush, subagent, background, skill_loop
  • Reflect max calls per session: 3
  • Sync roles: user
  • Flush every: 6 turns
  • Embedding model: BAAI/bge-small-en-v1.5 (384 dims)
  • Embeddings via API: false (local fastembed)

Database Stats (live, 2026-10-04)

Table Rows Notes
working_memory 773 Active memories
episodic_memory 1,047 Consolidated summaries
memory_embeddings 760 Working-tier embeddings
memory (legacy) 2,181 Old table, not migrated
vec_working / vec_working_rowids 760 Working vector index
vec_episodes / vec_episodes_rowids 1,035 Episodic vector index
vec_facts_rowids 0 memoria_facts is intentionally left unvectorised
Shared surface 0 Empty — cross-agent sharing not functional

Embedding coverage (live, 2026-10-04): 1,759/2,194 memories embedded (80.2%) — working 726/752 (96.5%), episodic 1,033/1,039 (99.4%); 13 embeddings generated in the previous 24h. Working coverage is above the 95% alert threshold enforced by the Embedding Coverage job (b63ed5817212, every 6h, Telegram topic 102).

Known Issues

⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x). working_memory and episodic_memory stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are not in memory, and the 23:30 sleep cycle consolidated 0 items. The health monitor ea79a2107e36 exited 1 at 2026-11-15 00:03:44 on STALE MEMORY PIPELINE (incident ea79a2_601037516abd, state: alerted, unclosed; its failure_type: auth is the scheduler's classifier, not evidence of a credential fault). Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing — it measured capture delay 29.4 min (healthy, exit 0, delivery suppressed), and the -589.5 min ℹ️ line belongs to the 00:03:35 run itself (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, -589.5 at line 1046, ❌ at 1047). mnemosyne_freshness_gate.py is magnitude-based (delay_min < -CAPTURE_LAG_MIN, default 120 min) but activity-aware: it can only fire once a conversational message postdates the last capture. The last captured message was the 09:03 AWST ios turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the 19:22 AWST desktop turn — after the 12:00 slot. The quiet was the 0 */12 cadence, not a sign-vs-magnitude bug. What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST, printed as (0 items) on every run yet tested only for age (> 36 h) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: consolidation_log.created_at mixes formats (recent rows +00:00, older rows naive AWST), so treating the column as one axis misplaces events by 8 h. The known stringified-config.yaml killer is ruled out (config.yaml mtime 2026-11-13 18:03; full YAML walk shows no [/{ scalars). Root cause not established — no error lines; the writes simply stop. Alert path verified working (delivery for_failure: 1, delivered 00:03:39). Update 2026-11-16 00:18 (next audit): the freeze is total, not degraded — working_memory 1130 / episodic_memory 1249, identical at every read of the stall, so 0 writes in 38 h 46 m; newest rows are still the [MODEL_REFRESH_PROPOSAL] maintenance writes at 09:32:58/09:32:59 and the newest user-turn capture is 2026-11-14 09:04:19. The monitor failed twice more (12:03:14, 2026-11-16 00:03:15) → cron_incidents 1 → 3 unclosed (141 rows total), each a single observation (first_seen_at == last_seen_at) because the signature embeds the elapsed-hours float (4.64157453 / 4.634034074444445 / 4.643481075555555), which is never equal twice — one durable fault, a fresh row every slot, nothing ever escalates. ⚠ And the ≈4.64 h in the alert is not a stall duration: mnemosyne_freshness_gate.py's newest_user_message() reports the age of the newest message whose session source is not cron/subagent — the Pulse/desktop turns, ~6 h cadence — so the alarm has understated severity on every fire (mapping table on the incident page). The consolidator's summary_preview now reads No-op consolidation cycle — timestamp updated for watchdog: it bumps its own timestamp to satisfy an age test, so a zero-item streak is invisible to an age-only gate; streak now ~74 cycles from 2026-11-14T02:04:42+00:00. Proposed, none applied: a zero-item-streak threshold, and normalising the value out of the incident signature. Update 2026-11-17 00:27 (third read): nothing has moved — working_memory 1130 / episodic_memory 1249, newest write 2026-11-14 09:32:59 AWST, so the stall is 62 h 54 m; unclosed alerted incidents 5 (143 rows total), consolidator zero-item streak 120 cycles, monitor failure_streak: 5. Update 2026-11-20 00:5x (fourth read) — the freeze ended by itself after 105 h 52 m, and the "capture dead" reading was wrong. working_memory 1130 → 1131, episodic_memory 1249 → 1250, newest write 2026-11-18 19:25:03 AWST (stored UTC 11:25:03) — the row is session_id: hermes_20261118_191808_5c9e08, content beginning [USER] You are helping me migrate contex…. Stall ran 105 h 52 m and ended with no intervention. ⚠ Correction: state.db lists only three non-cron sessions since 11-14 — two desktop on 11-14 (08:35/40 msgs, 08:52/41 msgs, both before the 09:33 boundary) and the 11-18 19:18 desktop (9 msgs, "Helping migrate context AI assistant") whose captured turn ends the freeze — so every interactive session in the window was captured, and the "stale pipeline" text was measuring conversational silence (the gate reads the age of the newest non-cron/subagent message). Still red on that same metric: monitor slots 11-18 00:01 and 12:01 failed, 11-19 00:01 completed, 11-19 12:01 and 11-20 00:01 failed — all STALE MEMORY PIPELINE, failure_streak: 2, incidents ea79a2_f2be2db3bfce (11-19 12:01:58) and ea79a2_aae4d914f3c9 (11-20 00:03:36), both alerted; their failure_type: auth is the scheduler's classifier, not a credential fault. Unclosed cron_incidents 5 → 4 (adds the audit's cbfe43_fa12cca1ad4b, auth, alerted_at: None). Consolidator newest row 2026-11-19T16:20:14+00:00 (running on ~31 min cadence), items_consolidated: 0, 199 of the last 200 cycles zero-item (was 120 at the 11-17 read). Zero-item-streak threshold still proposed, not applied. The watchdog's separate fault is the conflict queue — 59 unresolved vs a threshold of 50 → incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm. → incidents/2026-11-14-mnemosyne-capture-stall

⚠ Security - live .env values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST). 17 rows carry three live credentials - TELEGRAM_BOT_TOKEN (4 rows), MATTERMOST_TOKEN (6) and HERMES_CUSTOM_OMNIROUTE_20128_API_KEY (7) - across memories.content, working_memory.content, episodic_memory.content, gists.text, memoria_facts.context_snippet and the fts_episodes / fts_working mirrors (per-key breakdown on the incident page). The 0-byte episodic_memory.db and memoria_facts.db stubs beside mnemosyne.db hold nothing - those names are tables inside the main DB. Capture stores verbatim session text (the 1.7 GB session store state.db itself carries all four values), so any session in which a key was printed becomes recall material. Recall is the exposure path: these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed .db files under backups/) reproduce the whole DB. Not remediated. -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts. Capture-time redaction of .env values is the durable fix; rotation is the only remedy that also covers the existing copies.

Tier counts + pipeline status (live, 2026-10-18 12:1x — health job ea79a2107e36, exited 1): DB 27 MB; working 801, episodic 1,065; embeddings working 775/801 (96.8 %), episodic 1,059/1,065 (99.4 %) — both above the 95 % floor, no reindex needed. FTS5 aligned on all three indexes (fts_working / fts_episodes / fts_facts); consolidation ran 2026-10-18 03:55 (0 items — nothing new to consolidate); 1 unresolved conflict (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a capture-staleness false positive: working_memory had no non-cron row since 2026-10-16 11:07 simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. The 24 h threshold in ~/.hermes/scripts/mnemosyne-health-check.sh:92 will keep flagging idle weekends; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.

Tier counts (live, 2026-10-09 22:27): working 773, episodic 1,047, canonical_facts 97, facts 871, memoria_facts 407, memoria_preferences 132, conflicts 152 (1 open — id 152, contradiction, created 2026-10-09 13:11:38), working embeddings 760/773, episodic embeddings 1,043/1,047 (99.6%). Ranking relevance is the live issue, not coverage — see incidents/2026-10-09-memory-recall-relevance.

Tier counts (live, 2026-11-01 03:0x — the 2026-10-09 ranking incident is now RESOLVED): working 941, episodic 1,124, canonical_facts 90, conflicts 175 (11 open — ids 165–175, all contradiction, all created in one batch at 2026-10-31 19:26:39 and sharing a single fact_a_id, the extraction-noise signature), working embeddings 927/941 (98.5%), episodic embeddings 1,124/1,124 (100%) — both tiers above the 95% floor, no reindex needed. The recall weights are live in ~/.hermes/mnemosyne/config.yaml (vec_weight: 0.5, fts_weight: 0.3, importance_weight: 0.3, plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 in the environment) — not in ~/.hermes/config.yaml's memory.mnemosyne block, which is why the 2026-10-10 probe wrongly concluded the fix was never applied. The original regression query now returns only substantive current-state facts; the stale Uber Eats episode no longer outranks them → incidents/2026-10-09-memory-recall-relevance (resolved).

Tier counts (live, 2026-10-12 12:02 — health job ea79a2107e36, green): working 760/786 embedded (96.7%), episodic 1050/1056 (99.4%) — both above the 95% floor, no reindex needed (b63ed5817212 continues to backfill). Orphan embeddings 0. FTS5 exactly aligned: fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876. 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is not the live issue — ranking relevance is (see incidents/2026-10-09-memory-recall-relevance).

Tier counts (live, 2026-10-12 00:1x): working 786, episodic 1,056, facts 876 (fts_facts 876 — FTS aligned), fts_working 786, memoria_facts 410, conflicts 152. Embeddings: working 773/786 (98.3%), episodic 1,052/1,056 (99.6%) — both above the 95% reindex threshold; the coverage job b63ed5817212 backfilled 15 vectors since the 2026-10-11 00:08 check.

Capture clock is naive UTC — never compare it to AWST now (2026-10-11 false alarm). At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing working_memory MAX(created_at) = 2026-10-10 13:24:53 against the local clock. created_at is stored as a naive UTC string (the same trap as consolidation_log), so the real gap was 2.7 h: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — working_memory gained 2 rows on 2026-10-11 (11:19:37 and 11:51:39 UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST. 2026-11-14/15 recurrence, new flavour: the monitor's own escalation text printed last capture 2026-11-14 01:32:59 (raw UTC) for a write that landed at 09:32:59 AWST, so the alert reads as a ~23 h hole when the true gap was 14 h 31 m — the same trap, now inside the monitor's output rather than the audit's reading → incidents/2026-11-14-mnemosyne-capture-stall.

Vector embedding coverage gap — RESOLVED (2026-08-02)

  • Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
  • Resolved by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is 96.5% working / 99.4% episodic, and new embeddings are still being generated. The figures above are kept as the before-state only.
  • Standard repair if coverage drops below 95% working: ~/.hermes/hermes-agent/venv/bin/mnemosyne reindex --yes (the Embedding Coverage job b63ed5817212 triggers the same path every 6h).

Vector recall silently ran FTS-only (2026-10-03 — repaired)

  • fastembed was absent from every uv-managed runtime, so the dense leg of mnemosyne_recall produced no candidates and recall fell back to FTS5/keyword scoring without raising an error. Sessions kept working, which is why it went unnoticed.
  • Compounded by FASTEMBED_CACHE_PATH being unset in ~/.hermes/.env, and by the embedding worker sometimes launching in isolated (python -I) mode, which does not inherit PYTHONPATH.
  • Repaired 2026-10-03: fastembed installed into the bootstrap-selected runtime, FASTEMBED_CACHE_PATH pinned, and hermes-runtime-deps-guard.sh extended to assert fastembed + cache path + model dir every 15 min alongside its existing dotenv/mnemosyne checks.
  • CLI caveat — do not misread mnemosyne doctor. Doctor and the recall CLI run under the legacy venv/ (Python 3.11), where the sqlite_vec extension is not importable; doctor therefore prints vec0: present_but_unloadable and marks the vector tiers unavailable even while the gateway embeds normally. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py, not with doctor's vec0 line.
  • Benign noise: pthread_setaffinity_np failed … mask: {0,} from onnxruntime in mcp-stderr.log is a container affinity warning emitted on every model load — not an error.
  • See incidents/2026-10-03-mnemosyne-vector-recall-fts-only.

Profile isolation chosen over shared surface

|- The ops profile was enabled with mnemosyne on 2026-08-02 |- Decision: profile_isolation: true was set on ops, giving each profile its own independent memory bank — zero cross-contamination |- data/shared/mnemosyne.db exists with 0 rows; cross-profile sharing is intentionally unused |- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour |- Config in ~/.hermes/profiles/ops/config.yaml: memory.mnemosyne.profile_isolation: true, allow_tool_override: false, shared_surface_read: false |- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently

Legacy table not migrated

  • 2,181 rows in the old memory table are not in the new working_memory/episodic_memory system
  • These are only accessible via the legacy memory tool, not via mnemosyne_recall

Gateway notifications were captured as user memories (fixed 2026-09-30)

  • Hermes's background-process notifications (long-running terminal jobs reporting completion) were being written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
  • 36 such rows were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
  • Two canonical facts were also wrong because of the same class of drift: hermes_cron_model_default named the removed provider custom:omni instead of custom:omni-101, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
  • Rule: treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.

Known Management Jobs

  • Mnemosyne Auto-Consolidate every 30m — script: consolidate_mnemosyne.sh, mode: no-agent
  • Mnemosyne DB Backup weekly Sunday 03:00 — script: mnemosyne-backup.sh, mode: no-agent
  • Mnemosyne Health Monitor every 12h — script: mnemosyne-health-monitor.sh, mode: no-agent since the 2026-11-08 conversion (job ea79a2107e36), reports only on exception. ⚠ 2026-11-15 → 11-16: failing on a three-slot streak — last_status: error, failure_streak 3 (failed 00:00:53.928, 12:01:24.719, 2026-11-16 00:01:33.030), the fleet's only non-clean row. Each failure raises a new unclosed cron_incidents row for the same condition (signature fragmentation — see the capture-stall incident page). ⚠ The no-agent conversion was verified end-to-end on 2026-11-08: the 18:00 slot (agent mode) ran 12:00:51 → 18:00:42 = 5 h 59 m 51 s, and the converted job's first slot ran 2026-11-09 00:00:48 → 00:02:02 = 73 s, exit 0, silent — output/ea79a2107e36/2026-11-09_00-02-00.md is 159 bytes against 105,982 B for the last agent-mode file. Live read at 2026-11-09 00:0x: working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 % (so the < 95 % alert is correctly quiet), 1043 embeddings generated in 24 h. The script compares with awk, not bc — bc is not installed on this host (mnemosyne-health-monitor.sh:47), which is why the old monitor's arithmetic silently defaulted instead of failing loudly. ⚠ 2026-10-30: the monitor was twice flagged and neither flag survived contact with finished_at. Its 00:01:08 slot was auto-paused at 01:04:22 on an "execution unverified / gateway connection failed" admission (incidents/2026-10-30-mnemosyne-health-monitor-paused) — but the slot was still running and closed completed at 10:02:48 after 10 h 01 m 40 s, delivering normally and auto-closing incident ea79a2_d93c3acdc4f2; the job is enabled, last_status: ok, streak 0. Its 12:01 slot was then lost to the CT460 host outage (5 h 30 m dark, incidents/2026-10-30-ct460-host-outage-scheduler-gap) and fired in the 16:37:21 catch-up. Check finished_at before believing a long-running slot failed.
  • Mnemosyne Embedding Coverage every 6h — job b63ed5817212, alerts to Telegram topic 102 only when working-tier coverage drops below 95%
  • Mnemosyne Watchdog every 15m — script: mnemosyne-watchdog.py, mode: no-agent ⚠ 2026-11-15 11:31:20 slot failed after 17 m 49.8 s — runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds. The 11:45 slot was skipped and 12:01:25 completed, so it self-recovered with one slot lost. New signature for this job — the 2026-08-19 / 10-16 occurrences had the guard quoting another job's error. 1 failure in 97 rows across 11-15 → 11-16.
  • ⚠ Blast-radius note (2026-10-16). Both monitors above (ea79a2107e36 Health every 12 h, b63ed5817212 Embedding Coverage every 6 h) are pinned to custom:omni-101 — the same node the memory system's own reporting depends on. During the 2026-10-14 → 10-15 omniroute-1 outage both failed on every slot, so no coverage figures were refreshed for ~34 hours; the last explicit read remains the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %), and the 2026-10-16 00:09:48 Health Monitor run is the first clean read since. The two script-only jobs (consolidate_mnemosyne.sh, mnemosyne-backup.sh) were unaffected — they make no model call. Monitoring that shares a blast radius with its subject is degraded monitoring; a coverage alert that cannot fire during a router outage is exactly the case this has to cover. → incidents/2026-10-14-omniroute-node-connection-refused-storm

History and Evidence

  • 2026-10-03: vector recall found running FTS-only (fastembed missing from the managed runtimes); repaired the same day and the dependency guard extended — see above and incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
  • 2026-09-30 memory-hygiene pass: 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (hermes_cron_model_default → custom:omni-101, job-radar model). No unresolved conflicts left in the working tier.
  • 2026-08-19 Mnemosyne Watchdog incident: Watchdog job (96eb95ce12f2) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See systems/scheduled-tasks Known Issues section.
  • 2026-08-18 incident: Working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See incidents/2026-08-18-mnemosyne-capture-pipeline.
  • 2026-08-01 incident: Automatic conversational capture was broken by stringified structured YAML, especially sync_roles: "['user']". Fixed by rewriting with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
  • 2026-08-01 cleanup: Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
  • 2026-07-29 incident: memory.mnemosyne.tools: None caused schema loading warnings. Fixed by removing the line. See incidents/2026-07-29-mnemosyne-relay.
  • 2026-07-29 relay scope corruption: RuntimeError: scope handle is not at the top of the stack — 3 occurrences, self-resolved.

Re-verified 2026-10-10 (audit window; live probes 2026-10-11 00:14-00:20 AWST)

  • Gateway load path repaired. The gateway's activate_dependencies() selects one committed uv env (795020dbf24343cd9c52df24723dca83); it was missing mnemosyne / mnemosyne_hermes and the native dependencies. Repaired 2026-10-10 21:26-21:34 and confirmed live from 22:01:19 (registered (40 tools) -> activated). Detail: incidents/2026-09-28-hermes-managed-runtime-deps.
  • "Capture silent since 10-09 15:20" was a false alarm. It resolved to no interactive sessions in the window - the window was cron-only traffic. What the surrounding hours did capture was dominated by gateway background-process notifications stored as [USER] [Anthony Martin] [IMPORTANT: Background process ...] rows: recall-visible, low signal. A capture filter would help; no fix applied (carried open item).
  • Coverage script executable again: ~/.hermes/scripts/check_embedding_coverage.py now carries mode 0755, so the monitor's non-executable finding is resolved.
  • Open, unverified: 13 orphan rows in memory_embeddings (oldest 2026-08-22) reported by a 2026-10-10 monitor run - neither confirmed nor cleaned by this audit.

Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST)

  • Both monitors rode the omniroute-1 503 window; both are green again. ea79a2107e36 (health, every 12 h) failed 10-17 00:00 with RuntimeError: HTTP 503 ... resource pressure and delivered its alert; b63ed5817212 (coverage, every 6 h) failed 10-16 12:03 the same way. Since 10-17 12:00 every slot of both has completed with delivery_outcome: suppressed (nothing to report) — including 10-18 00:00.
  • No fresh coverage figures exist in the ledger. These jobs report to Telegram, not to the execution row, so the last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %), both above the 95 % reindex threshold. Nothing here indicates a reindex is due.
  • The watchdog triple-checks the same thing. 96eb95ce12f2 (every 15 min) exited 1 four times — 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the coverage monitor's own 503 error, i.e. one sentinel producing four failure rows. Read failures per job, not per row. It has been clean on every slot since (latest 10-18 00:30).

Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST)

  • Cold start is the new failure vector for consumers: 10–45 s. The Conduit voice plugin (conduit_push) gave one request — provider start included — a hard-coded 10 s budget (MEMORY_TIMEOUT_S). Mnemosyne's cold start for the conduit-voice session regularly exceeds that, so three calls were abandoned by the memory layer (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call), and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default 60 s, overridable with CONDUIT_MEMORY_TIMEOUT_S, plus a TokenError guard that logs and proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists after it. → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit
  • Rule for other consumers: a 10 s memory budget is below the observed cold-start floor. Any new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall" rather than fail. The underlying cold start was not optimised — only the budget was raised.
  • Capture and coverage unchanged from the 10-18 read (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.