Files
agent-estate-wiki/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
T
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

5.6 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST) incident active 2026-11-20 2026-11-20 2026-11-20 2026-11-21 high
mnemosyne
memory
conflicts
watchdog
cron
alert-fatigue
incident
live-cron-executions-db
live-cron-incidents
live-mnemosyne-db
live-deliveries-db
live-jobs-json

Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)

Open. 96eb95ce12f2 (Mnemosyne Watchdog, 15-minute cadence) has failed every slot since 2026-11-18 20:01:06 AWST — 115 consecutive failures through 2026-11-20 00:31:16 AWST, the last recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit). cron/jobs.json reads last_status: error, failure_streak: 115.

This is the estate's only red cron row with a large streak, and it is a single durable condition re-reported every 15 minutes, not a series of separate faults.

The failure is a fixed, self-describing threshold trip

Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)

cron_incidents row 96eb95_98ade422d195 — state: alerted, failure_type: script, first_seen_at 2026-11-18T20:01:15.906837+08:00, last_seen_at 2026-11-20T00:31:16.326357+08:00, alerted_at 2026-11-19T20:46:23.428143+08:00, closed_at: None, output file /home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md.

Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per slot → incidents/2026-11-14-mnemosyne-capture-stall), this one fixes on the constant count 59 and therefore stays a single row. The cost of that stability is the mirror image: it never closes, and it re-alerts forever.

Alerting works — and that is the problem

deliveries.db, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:

delivered execution
2026-11-18 20:01:16 f9865723b4f64d9fbfca7711ab053056
2026-11-19 02:16:16 7aa35681fd914a1e8db64b8dd0e09c84
2026-11-19 08:31:17 1ccc7e280d234287973edf5e0b35a5b7
2026-11-19 12:01:59 396940d9514b419eb024b83b3c10198d
2026-11-19 14:31:35 0766953f92dc4fc4b0d0404dca51683f
2026-11-19 20:46:19 db010a8294bf4cd78457a265a5fb167c
2026-11-20 00:03:37 64c6b5202463487c906f860bb746138b

All status: delivered, all for_failure: 1. So detection, the failure notice and the push path are all working. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days — which is exactly how a red signal stops meaning anything: while this row is red, a new Mnemosyne fault arriving on the same job is indistinguishable from the standing one.

What the 59 conflicts actually are

Live read of ~/.hermes/mnemosyne/data/mnemosyne.db (2026-11-20 00:3x AWST): 223 conflicts total, 59 unresolved. Unresolved by creation day:

created unresolved
2026-10-31 11
2026-11-02 2
2026-11-03 12
2026-11-06 1
2026-11-07 2
2026-11-08 1
2026-11-12 13
2026-11-13 3
2026-11-18 14
total 59

Two things stand out:

  1. Most of the backlog is old. 42 of the 59 predate 2026-11-14; the oldest batch is from 2026-10-31, which is the same batch recorded on systems/mnemosyne as "all created in one batch at 2026-10-31 19:26:39 and sharing a single fact_a_id, the extraction-noise signature". The documented remediation for this class is to bulk-resolve them as false_positive (see the mnemosyne-operations skill) — not applied here.
  2. The threshold has been raised before. The 2026-10-18 and 2026-11-01 reads on systems/mnemosyne both record "1 unresolved conflict (benign, threshold 10)". It now reads 50. So the tripwire was already loosened 5×, and the backlog has since grown past even that. UNVERIFIED: when the threshold was changed from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was not opened in this window.

The 2026-11-18 batch coincides with the migration session

The 14 conflicts created on 2026-11-18 are the newest batch, and 2026-11-18 is also the day of the only interactive session in this window — 20261118_191808_5c9e08 (desktop, 9 messages, title "Helping migrate context AI assistant"), whose turns were captured into working_memory at 19:25:03 AWST and summarised episodically at 19:55:20 AWST. Consistent with the standing assessment that most conflicts in this bank are LLM-extraction artefacts rather than real contradictions, but UNVERIFIED: that the 14 rows are extraction noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis — the fact pairs were not inspected.

Not applied / open

  • Nothing was resolved, reclassified or deleted. No conflict row was touched; no threshold was changed.
  • The watchdog's cadence, signature construction and alert path were left as-is.
  • Open question: should a standing threshold breach alert on state change rather than on every slot? Same family as the zero-item-consolidation streak problem on incidents/2026-11-14-mnemosyne-capture-stall — both are conditions that re-report continuously because the gate tests a level, not a change.

→ systems/mnemosyne, systems/scheduled-tasks