- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None) - new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50) - corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured) - systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed) - index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
112 lines
5.6 KiB
Markdown
112 lines
5.6 KiB
Markdown
---
|
||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||
type: incident
|
||
status: active
|
||
created: 2026-11-20
|
||
updated: 2026-11-20
|
||
verified_on: 2026-11-20
|
||
review_after: 2026-11-21
|
||
confidence: high
|
||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||
---
|
||
|
||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||
|
||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||
|
||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||
re-reported every 15 minutes, not a series of separate faults.
|
||
|
||
## The failure is a fixed, self-describing threshold trip
|
||
|
||
```
|
||
Script exited with code 1
|
||
stdout:
|
||
🚨 Many unresolved conflicts: 59 (threshold: 50)
|
||
```
|
||
|
||
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
|
||
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
|
||
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
|
||
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
|
||
|
||
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
|
||
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
|
||
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
|
||
re-alerts forever.
|
||
|
||
## Alerting works — and that is the problem
|
||
|
||
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
|
||
|
||
| delivered | execution |
|
||
|---|---|
|
||
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
|
||
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
|
||
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
|
||
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
|
||
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
|
||
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
|
||
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
|
||
|
||
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
|
||
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
|
||
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
|
||
arriving on the same job is indistinguishable from the standing one.
|
||
|
||
## What the 59 conflicts actually are
|
||
|
||
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
|
||
unresolved.** Unresolved by creation day:
|
||
|
||
| created | unresolved |
|
||
|---|---|
|
||
| 2026-10-31 | 11 |
|
||
| 2026-11-02 | 2 |
|
||
| 2026-11-03 | 12 |
|
||
| 2026-11-06 | 1 |
|
||
| 2026-11-07 | 2 |
|
||
| 2026-11-08 | 1 |
|
||
| 2026-11-12 | 13 |
|
||
| 2026-11-13 | 3 |
|
||
| **2026-11-18** | **14** |
|
||
| **total** | **59** |
|
||
|
||
Two things stand out:
|
||
|
||
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
|
||
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
|
||
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
|
||
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
|
||
applied here**.
|
||
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
|
||
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
|
||
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
|
||
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
|
||
not opened in this window.
|
||
|
||
## The 2026-11-18 batch coincides with the migration session
|
||
|
||
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
|
||
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
|
||
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
|
||
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
|
||
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
|
||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||
the fact pairs were not inspected.
|
||
|
||
## Not applied / open
|
||
|
||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||
- The watchdog's cadence, signature construction and alert path were left as-is.
|
||
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
|
||
Same family as the zero-item-consolidation streak problem on
|
||
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
|
||
the gate tests a level, not a change.
|
||
|
||
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|