wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None) - new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50) - corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured) - systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed) - index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
This commit is contained in:
@@ -1,17 +1,21 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled for 62+ hours (2026-11-14 09:33 AWST onward) — five unclosed alerts, freeze total on re-read
|
||||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-18
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled for 62+ hours (from 2026-11-14 09:33 AWST)
|
||||
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
|
||||
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
|
||||
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
|
||||
> not a broken hook. Read the newest re-read section before the earlier readings.
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
@@ -144,6 +148,58 @@ total.**
|
||||
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||
|
||||
|
||||
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
|
||||
|
||||
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
|
||||
|
||||
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
|
||||
| --- | --- | --- |
|
||||
| `working_memory` | 1130 | **1131** |
|
||||
| `episodic_memory` | 1249 | **1250** |
|
||||
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
|
||||
|
||||
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
|
||||
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
|
||||
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
|
||||
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
|
||||
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
|
||||
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
|
||||
|
||||
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
|
||||
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
|
||||
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
|
||||
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
|
||||
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
|
||||
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
|
||||
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
|
||||
a broken capture path — is the durable defect on this page.
|
||||
|
||||
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
|
||||
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
|
||||
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
|
||||
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
|
||||
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
|
||||
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
|
||||
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
|
||||
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
|
||||
|
||||
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
|
||||
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
|
||||
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
|
||||
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
|
||||
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
|
||||
applied**.
|
||||
|
||||
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
|
||||
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
|
||||
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
|
||||
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
|
||||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
|
||||
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
|
||||
|
||||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||
---
|
||||
|
||||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||
|
||||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||
|
||||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||
re-reported every 15 minutes, not a series of separate faults.
|
||||
|
||||
## The failure is a fixed, self-describing threshold trip
|
||||
|
||||
```
|
||||
Script exited with code 1
|
||||
stdout:
|
||||
🚨 Many unresolved conflicts: 59 (threshold: 50)
|
||||
```
|
||||
|
||||
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
|
||||
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
|
||||
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
|
||||
|
||||
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
|
||||
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
|
||||
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
|
||||
re-alerts forever.
|
||||
|
||||
## Alerting works — and that is the problem
|
||||
|
||||
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
|
||||
|
||||
| delivered | execution |
|
||||
|---|---|
|
||||
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
|
||||
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
|
||||
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
|
||||
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
|
||||
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
|
||||
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
|
||||
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
|
||||
|
||||
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
|
||||
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
|
||||
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
|
||||
arriving on the same job is indistinguishable from the standing one.
|
||||
|
||||
## What the 59 conflicts actually are
|
||||
|
||||
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
|
||||
unresolved.** Unresolved by creation day:
|
||||
|
||||
| created | unresolved |
|
||||
|---|---|
|
||||
| 2026-10-31 | 11 |
|
||||
| 2026-11-02 | 2 |
|
||||
| 2026-11-03 | 12 |
|
||||
| 2026-11-06 | 1 |
|
||||
| 2026-11-07 | 2 |
|
||||
| 2026-11-08 | 1 |
|
||||
| 2026-11-12 | 13 |
|
||||
| 2026-11-13 | 3 |
|
||||
| **2026-11-18** | **14** |
|
||||
| **total** | **59** |
|
||||
|
||||
Two things stand out:
|
||||
|
||||
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
|
||||
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
|
||||
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
|
||||
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
|
||||
applied here**.
|
||||
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
|
||||
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
|
||||
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
|
||||
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
|
||||
not opened in this window.
|
||||
|
||||
## The 2026-11-18 batch coincides with the migration session
|
||||
|
||||
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
|
||||
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
|
||||
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
|
||||
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
|
||||
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
|
||||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||
the fact pairs were not inspected.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||
- The watchdog's cadence, signature construction and alert path were left as-is.
|
||||
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
|
||||
Same family as the zero-item-consolidation streak problem on
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
|
||||
the gate tests a level, not a change.
|
||||
|
||||
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||
---
|
||||
|
||||
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
|
||||
|
||||
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
|
||||
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
|
||||
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
|
||||
reads `running` (started `2026-11-20T00:00:08`).
|
||||
|
||||
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
|
||||
|
||||
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
|
||||
|
||||
| slot | started | finished / last seen | duration | status |
|
||||
|---|---|---|---|---|
|
||||
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
|
||||
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
|
||||
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
|
||||
|
||||
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
|
||||
|
||||
```
|
||||
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
|
||||
make this request. Please purchase more credits to continue using the service. (HTTP 400);
|
||||
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
|
||||
openai-compati…
|
||||
```
|
||||
|
||||
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
|
||||
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
|
||||
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
|
||||
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
|
||||
|
||||
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
|
||||
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
|
||||
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
|
||||
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
|
||||
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
|
||||
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
|
||||
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
|
||||
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
|
||||
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
|
||||
So the outage produced **no push to Anthony** and the red cron row was the only signal.
|
||||
|
||||
## Why this matters beyond one job
|
||||
|
||||
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
|
||||
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
|
||||
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
|
||||
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
|
||||
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
|
||||
|
||||
## Precedent — and a repeating pattern
|
||||
|
||||
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
|
||||
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
|
||||
|
||||
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
|
||||
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
|
||||
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
|
||||
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||
rather than at the alert throttle.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||
cron entry was edited.
|
||||
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
|
||||
entry appended to `log.md` the same night).
|
||||
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
|
||||
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
|
||||
basis: the truncated 314-char error string.
|
||||
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
|
||||
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
|
||||
Reference in New Issue
Block a user