Files
agent-estate-wiki/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00

112 lines
5.6 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
## The failure is a fixed, self-describing threshold trip
```
Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)
```
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
re-alerts forever.
## Alerting works — and that is the problem
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
| delivered | execution |
|---|---|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
arriving on the same job is indistinguishable from the standing one.
## What the 59 conflicts actually are
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
unresolved.** Unresolved by creation day:
| created | unresolved |
|---|---|
| 2026-10-31 | 11 |
| 2026-11-02 | 2 |
| 2026-11-03 | 12 |
| 2026-11-06 | 1 |
| 2026-11-07 | 2 |
| 2026-11-08 | 1 |
| 2026-11-12 | 13 |
| 2026-11-13 | 3 |
| **2026-11-18** | **14** |
| **total** | **59** |
Two things stand out:
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
applied here**.
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
not opened in this window.
## The 2026-11-18 batch coincides with the migration session
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
- The watchdog's cadence, signature construction and alert path were left as-is.
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
Same family as the zero-item-consolidation streak problem on
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
the gate tests a level, not a change.
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]