wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)

This commit is contained in:
Tony0410
2026-11-21 00:40:00 +08:00
parent f603c29d17
commit ca0ff5aa80
11 changed files with 484 additions and 31 deletions
@@ -1,13 +1,13 @@
---
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
title: Mnemosyne capture is dead for the long-running desktop DM session since 2026-11-13 13:10 AWST — ~57 uncaptured turns, four unclosed health-monitor alerts
type: incident
status: active
created: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-11-22
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, session-scoped, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
@@ -200,6 +200,47 @@ silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98a
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
### Fifth read — 2026-11-21 00:0x–00:5x AWST: **the 11-20 conclusion above is withdrawn.** Capture is not "quiet", it is **dead for one specific session**, and the boundary is exact.
The 11-20 read counted **sessions** in `state.db`, found only three in the window, and concluded that every
interactive session had been captured — so the alert was measuring conversational silence. **The wrong unit.**
Capture is per *message*. Counted per message, the picture inverts:
| | |
|---|---|
| Rows this session owns in `working_memory` | **124** (84 under the bare id `hermes_20261031_190813_95fbde`, 40 under `hermes_agent:main:local:dm:20261031_190813_95fbde`) |
| Their span | **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC = 2026-11-13 13:10:35 AWST** — then nothing |
| Last *captured* turn in `state.db` | **2026-11-13 13:06:05 AWST** (Pulse report `pc36e5568`, row written 13:10:35) |
| First *uncaptured* turn | **2026-11-13 13:21:00 AWST** (`Pulse delivery check p65be4717`) — and every turn since |
| Uncaptured turns since | **~57** (11-13 19:21 → 11-21 00:00): Pulse reports, delivery checks, morning briefs, `[System: …]` notices |
| Writes from this session after that boundary | **0** |
- **The break sits in a 15-minute gap, and the 11-13 13:14 gateway restart sits inside it.** Last captured turn
13:06:05, first uncaptured 13:21:00; the 11-13/11-14 audit record documents a **gateway restart at 13:14**
that day (the one that cleared the stale `TELEGRAM_HOME_CHANNEL_THREAD_ID`). *Leading hypothesis:* the
restart left this session's capture watermark at-or-past its tip, so the session syncs nothing while new
sessions start clean. **NOT PROVEN** — the oldest surviving `agent.log` file begins **2026-11-13 19:03:34**,
so the 13:14 window has rotated out of the log horizon and no startup banner is recoverable.
- **Capture is not globally dead — it works for *new* sessions.** The single write after that boundary,
`2026-11-18 11:25:03 UTC = 19:25:03 AWST`, came from `hermes_20261118_191808_5c9e08`, a session created by a
genuine prompt that evening. That is the **only** write in eight days.
- **The health monitor already says exactly this,** in its own words: `ea79a2107e36` is failing with
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`. The gate was right; the 11-20
reading of it was wrong.
**Corrected statement for the record:**
> **Mnemosyne has captured nothing from the desktop DM session `20261031_190813_95fbde` since 2026-11-13
> 13:10:35 AWST — ~8 days and ~57 user-role turns — while capturing normally from sessions created after that
> date.**
**Counters (11-21 read):** `working_memory` **1,131** / `episodic_memory` **1,250** — both **unchanged** from
the 11-20 read; global silence since the last write **52 h 35 m**. **Unclosed incidents: 4 → 5**
(`cbfe43_fa12cca1ad4b` auth, `alerted_at: None`; `96eb95_98ade422d195` script, still alerting 11-21 00:31:16 (00:16:15 at the first read);
`ea79a2_f2be2db3bfce` 11-19 12:01:58; `ea79a2_aae4d914f3c9` 11-20 00:03:36; **new** `ea79a2_ace0f263e18a`
11-21 00:03:12). **Next probe:** replay one Pulse-shaped turn into that session with the provider's debug
logging on, and diff against a pre-11-13 capture in `agent.log`.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
@@ -1,22 +1,22 @@
---
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
title: Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-11-22
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
# Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`** *(that is the **2026-11-20** reading — this page was written then; the re-read below carries the streak to **210**, `last_seen_at` **2026-11-21 00:31:16**)*.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
@@ -99,6 +99,22 @@ LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** th
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; **no signature fragmentation**
- `failure_streak` = **210** (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is
11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red).
- **Incident `96eb95_98ade422d195` is one row, not a series:** `first_seen_at` **2026-11-18 20:01:15**,
`last_seen_at` **2026-11-21 00:31:16 (00:16:15 at the first read)**, `alerted_at` **2026-11-20 21:31:24**, `closed_at` empty. The 11-20
page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication **holds**.
- **The contrast is the health monitor**, which re-forms a **new incident id every slot** for one standing
condition (`ea79a2_f2be2db3bfce` 11-19 12:01:58 → `ea79a2_aae4d914f3c9` 11-20 00:03:36 →
`ea79a2_ace0f263e18a` 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when
reading the ledger for "new" versus "chronic".
- Error text unchanged: **59 unresolved conflicts against a threshold of 50**. Nothing was resolved,
reclassified or deleted at this read either.
- **Still the standing question:** a *level*-based test re-reports on every slot. A *state-change* alert would
have fired once on 11-18 20:01 and then stayed quiet.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
@@ -1,11 +1,11 @@
---
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
type: incident
status: active
status: resolved
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-11-28
confidence: high
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
@@ -74,6 +74,31 @@ skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
rather than at the alert throttle.
## Re-read 2026-11-21 00:0x AWST — **resolved**; and the ledger shows the outage was two nights, not one
- **The job is clean again.** `cbfe434a3b5e` reads `last_status: ok`, `failure_streak = 0`. The execution ledger
dates the outage exactly:
| Slot | Execution | Outcome |
|---|---|---|
| 2026-11-18 00:00:08 | `d6323ba3b7ca4d8a95d0f57bf20b4807` | **failed** (`delivery_outcome: suppressed`) |
| 2026-11-19 00:00:16 | `668984226ccc47b09cb66fe65ae89ddc` | **failed** (`delivery_outcome: suppressed`) |
| 2026-11-20 00:00:08 | `b88bd7d8007248399f57cccfc98b536a` | **completed** (report not delivered) |
| 2026-11-21 00:00:08 | `ec65ca362c6847fcb9405daa15d1f6ed` | running — the run authoring this note |
- **Two nights were lost, not one.** The 11-18 and 11-19 runs both failed, so windows **11-17 and 11-18** have
no audit of their own. The **11-20 run absorbed both** via its three-window catch-up (2026-11-17 → 2026-11-20),
read from the cron ledger and `state.db` rather than from conversation — `state.db` holds **no** interactive
sessions in those windows, so there was no conversation to miss.
- **No re-run of either lost slot was forced**; recovery came from the next scheduled slot running normally.
- **Not re-tested:** whether the credit/auth fault can recur. The job's pin, provider and schedule are unchanged
and no credential was rotated, so a recurrence would present identically.
- **The silent-skip class this page identified is now observed, not inferred:** both failed nights carry
`delivery_outcome: suppressed` and the incident row `cbfe43_fa12cca1ad4b` has `alerted_at: NULL` — two
consecutive failed nights produced **no alert and no report**, which is why the outage only surfaced when the
11-20 audit read the ledger. Whether that suppression is job-level config or framework behaviour is still
**UNVERIFIED** (the job's delivery config was not read).
## Not applied / open
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
@@ -0,0 +1,64 @@
---
title: Two digests died on `content_policy_blocked` (2026-11-20) — first content-filter failures on record
type: incident
status: active
created: 2026-11-21
updated: 2026-11-21
verified_on: 2026-11-21
review_after: 2026-11-22
confidence: high
tags: [cron, content-policy, provider-error, gmail, freshrss, digests, incident, unclassified-failure]
sources: [live-probe-cron-executions-db-2026-11-21, live-probe-cron-incidents-2026-11-21, live-probe-jobs-json-2026-11-21]
---
# Two digests died on `content_policy_blocked` (2026-11-20)
**The first content-filter failures in the retained log.** Two unrelated jobs, on two different feeds, died
with the *identical* terminal string on 2026-11-20 — hours apart, with every other job in the window clean.
## What happened
| Job | Job ID | Slot (AWST) | Execution | Duration | Terminal error |
|---|---|---|---|---|---|
| Gmail Action Digest | `08eb86f02d37` | 08:31:06 → 08:32:45 | `a490b74a509e4303acad671ee1ee80cf` | 1m39s | `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | 12:01:14 → 12:05:04 | `0e1c11f49313428799de604afe73595c` | 3m50s | same string |
Both jobs are pinned `provider: custom:omni-101` / `model: smart-route`. Both carry
**`failure_streak = 1`** — i.e. **isolated single failures, not chronic**: the preceding slot of each job
completed. FreshRSS Midday ran clean on 11-19 12:01.
## What is established
- The failure is raised **by the model-facing API, not by the job logic**. The string is a provider content
filter surfaced to the job as a wrapped `RuntimeError`; the digest scripts assembled their input fine.
- **Both jobs alerted, and both alerts are unclosed.** Incident rows:
- `08eb86_c0cca1a55b96` — first seen 11-20 08:32:44, `alerted_at` 08:32:46
- `ecc1fc_5a10b70830d1` — first seen 11-20 12:05:03, `alerted_at` 12:05:05
This is the **opposite** of [[incidents/2026-11-18-wiki-audit-credit-outage]], which failed silently for two
nights: here the alerting path worked end to end.
- `failure_type` is recorded as **`unknown`** for both. The incident classifier does not recognise this
class, so it can neither group nor auto-close it — a recurrence will alert, but will not be understood.
- Both left a full transcript: `~/.hermes/cron/output/08eb86f02d37/2026-11-20_08-32-44.md` and
`~/.hermes/cron/output/ecc1fc13655a/2026-11-20_12-05-03.md`.
## What is NOT established
- **Whether the trigger was the request or the response.** The message reads "Output data may contain
inappropriate content", but a filter that scans a request payload reports the same way. The Gmail digest
reads mail; the FreshRSS digest reads news — the shared factor is more likely a *token* the router's filter
dislikes than the subject matter. **The two output files above settle this**: diff what the two prompts had
in common.
- **Whether the router (`omni-101`) applies a filter its upstreams do not.** If it does, the same feeds can
fail again with nothing changed on this host. Probe: replay one of the two prompts against the router and
against an upstream provider directly, and compare.
- Whether it recurs. Two slots, two jobs, one day is a pattern worth watching — not yet a trend.
## Standing risk
Two of the estate's daily digests sit behind a provider filter that has already rejected them once, and the
classifier files the failure as `unknown` — so a recurrence will alert without being recognised. Cheapest
mitigation if it recurs: **no content-policy string is worth a lost digest** — route the two digest jobs to a
provider without the filter, or strip the highest-risk section from the prompt payload.
→ [[systems/scheduled-tasks]], [[systems/mnemosyne]]