Files
agent-estate-wiki/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
T

128 lines
7.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-11-22
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`** *(that is the **2026-11-20** reading — this page was written then; the re-read below carries the streak to **210**, `last_seen_at` **2026-11-21 00:31:16**)*.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
## The failure is a fixed, self-describing threshold trip
```
Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)
```
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
re-alerts forever.
## Alerting works — and that is the problem
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
| delivered | execution |
|---|---|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
arriving on the same job is indistinguishable from the standing one.
## What the 59 conflicts actually are
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
unresolved.** Unresolved by creation day:
| created | unresolved |
|---|---|
| 2026-10-31 | 11 |
| 2026-11-02 | 2 |
| 2026-11-03 | 12 |
| 2026-11-06 | 1 |
| 2026-11-07 | 2 |
| 2026-11-08 | 1 |
| 2026-11-12 | 13 |
| 2026-11-13 | 3 |
| **2026-11-18** | **14** |
| **total** | **59** |
Two things stand out:
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
applied here**.
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
not opened in this window.
## The 2026-11-18 batch coincides with the migration session
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; **no signature fragmentation**
- `failure_streak` = **210** (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is
11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red).
- **Incident `96eb95_98ade422d195` is one row, not a series:** `first_seen_at` **2026-11-18 20:01:15**,
`last_seen_at` **2026-11-21 00:31:16 (00:16:15 at the first read)**, `alerted_at` **2026-11-20 21:31:24**, `closed_at` empty. The 11-20
page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication **holds**.
- **The contrast is the health monitor**, which re-forms a **new incident id every slot** for one standing
condition (`ea79a2_f2be2db3bfce` 11-19 12:01:58 → `ea79a2_aae4d914f3c9` 11-20 00:03:36 →
`ea79a2_ace0f263e18a` 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when
reading the ledger for "new" versus "chronic".
- Error text unchanged: **59 unresolved conflicts against a threshold of 50**. Nothing was resolved,
reclassified or deleted at this read either.
- **Still the standing question:** a *level*-based test re-reports on every slot. A *state-change* alert would
have fired once on 11-18 20:01 and then stayed quiet.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
- The watchdog's cadence, signature construction and alert path were left as-is.
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
Same family as the zero-item-consolidation streak problem on
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
the gate tests a level, not a change.
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]