Files
agent-estate-wiki/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
T

7.0 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST) incident active 2026-11-20 2026-11-21 2026-11-21 2026-11-22 high
mnemosyne
memory
conflicts
watchdog
cron
alert-fatigue
incident
live-cron-executions-db
live-cron-incidents
live-mnemosyne-db
live-deliveries-db
live-jobs-json

Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read

Open. 96eb95ce12f2 (Mnemosyne Watchdog, 15-minute cadence) has failed every slot since 2026-11-18 20:01:06 AWST — 115 consecutive failures through 2026-11-20 00:31:16 AWST, the last recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit). cron/jobs.json reads last_status: error, failure_streak: 115 (that is the 2026-11-20 reading — this page was written then; the re-read below carries the streak to 210, last_seen_at 2026-11-21 00:31:16).

This is the estate's only red cron row with a large streak, and it is a single durable condition re-reported every 15 minutes, not a series of separate faults.

The failure is a fixed, self-describing threshold trip

Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)

cron_incidents row 96eb95_98ade422d195 — state: alerted, failure_type: script, first_seen_at 2026-11-18T20:01:15.906837+08:00, last_seen_at 2026-11-20T00:31:16.326357+08:00, alerted_at 2026-11-19T20:46:23.428143+08:00, closed_at: None, output file /home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md.

Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per slot → incidents/2026-11-14-mnemosyne-capture-stall), this one fixes on the constant count 59 and therefore stays a single row. The cost of that stability is the mirror image: it never closes, and it re-alerts forever.

Alerting works — and that is the problem

deliveries.db, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:

delivered execution
2026-11-18 20:01:16 f9865723b4f64d9fbfca7711ab053056
2026-11-19 02:16:16 7aa35681fd914a1e8db64b8dd0e09c84
2026-11-19 08:31:17 1ccc7e280d234287973edf5e0b35a5b7
2026-11-19 12:01:59 396940d9514b419eb024b83b3c10198d
2026-11-19 14:31:35 0766953f92dc4fc4b0d0404dca51683f
2026-11-19 20:46:19 db010a8294bf4cd78457a265a5fb167c
2026-11-20 00:03:37 64c6b5202463487c906f860bb746138b

All status: delivered, all for_failure: 1. So detection, the failure notice and the push path are all working. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days — which is exactly how a red signal stops meaning anything: while this row is red, a new Mnemosyne fault arriving on the same job is indistinguishable from the standing one.

What the 59 conflicts actually are

Live read of ~/.hermes/mnemosyne/data/mnemosyne.db (2026-11-20 00:3x AWST): 223 conflicts total, 59 unresolved. Unresolved by creation day:

created unresolved
2026-10-31 11
2026-11-02 2
2026-11-03 12
2026-11-06 1
2026-11-07 2
2026-11-08 1
2026-11-12 13
2026-11-13 3
2026-11-18 14
total 59

Two things stand out:

  1. Most of the backlog is old. 42 of the 59 predate 2026-11-14; the oldest batch is from 2026-10-31, which is the same batch recorded on systems/mnemosyne as "all created in one batch at 2026-10-31 19:26:39 and sharing a single fact_a_id, the extraction-noise signature". The documented remediation for this class is to bulk-resolve them as false_positive (see the mnemosyne-operations skill) — not applied here.
  2. The threshold has been raised before. The 2026-10-18 and 2026-11-01 reads on systems/mnemosyne both record "1 unresolved conflict (benign, threshold 10)". It now reads 50. So the tripwire was already loosened 5×, and the backlog has since grown past even that. UNVERIFIED: when the threshold was changed from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was not opened in this window.

The 2026-11-18 batch coincides with the migration session

The 14 conflicts created on 2026-11-18 are the newest batch, and 2026-11-18 is also the day of the only interactive session in this window — 20261118_191808_5c9e08 (desktop, 9 messages, title "Helping migrate context AI assistant"), whose turns were captured into working_memory at 19:25:03 AWST and summarised episodically at 19:55:20 AWST. Consistent with the standing assessment that most conflicts in this bank are LLM-extraction artefacts rather than real contradictions, but UNVERIFIED: that the 14 rows are extraction noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis — the fact pairs were not inspected.

Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; no signature fragmentation

  • failure_streak = 210 (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is 11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red).
  • Incident 96eb95_98ade422d195 is one row, not a series: first_seen_at 2026-11-18 20:01:15, last_seen_at 2026-11-21 00:31:16 (00:16:15 at the first read), alerted_at 2026-11-20 21:31:24, closed_at empty. The 11-20 page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication holds.
  • The contrast is the health monitor, which re-forms a new incident id every slot for one standing condition (ea79a2_f2be2db3bfce 11-19 12:01:58 → ea79a2_aae4d914f3c9 11-20 00:03:36 → ea79a2_ace0f263e18a 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when reading the ledger for "new" versus "chronic".
  • Error text unchanged: 59 unresolved conflicts against a threshold of 50. Nothing was resolved, reclassified or deleted at this read either.
  • Still the standing question: a level-based test re-reports on every slot. A state-change alert would have fired once on 11-18 20:01 and then stayed quiet.

Not applied / open

  • Nothing was resolved, reclassified or deleted. No conflict row was touched; no threshold was changed.
  • The watchdog's cadence, signature construction and alert path were left as-is.
  • Open question: should a standing threshold breach alert on state change rather than on every slot? Same family as the zero-item-consolidation streak problem on incidents/2026-11-14-mnemosyne-capture-stall — both are conditions that re-report continuously because the gate tests a level, not a change.

→ systems/mnemosyne, systems/scheduled-tasks