wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)
This commit is contained in:
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
title: Mnemosyne capture is dead for the long-running desktop DM session since 2026-11-13 13:10 AWST — ~57 uncaptured turns, four unclosed health-monitor alerts
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, session-scoped, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
@@ -200,6 +200,47 @@ silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98a
|
||||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||
|
||||
### Fifth read — 2026-11-21 00:0x–00:5x AWST: **the 11-20 conclusion above is withdrawn.** Capture is not "quiet", it is **dead for one specific session**, and the boundary is exact.
|
||||
|
||||
The 11-20 read counted **sessions** in `state.db`, found only three in the window, and concluded that every
|
||||
interactive session had been captured — so the alert was measuring conversational silence. **The wrong unit.**
|
||||
Capture is per *message*. Counted per message, the picture inverts:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Rows this session owns in `working_memory` | **124** (84 under the bare id `hermes_20261031_190813_95fbde`, 40 under `hermes_agent:main:local:dm:20261031_190813_95fbde`) |
|
||||
| Their span | **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC = 2026-11-13 13:10:35 AWST** — then nothing |
|
||||
| Last *captured* turn in `state.db` | **2026-11-13 13:06:05 AWST** (Pulse report `pc36e5568`, row written 13:10:35) |
|
||||
| First *uncaptured* turn | **2026-11-13 13:21:00 AWST** (`Pulse delivery check p65be4717`) — and every turn since |
|
||||
| Uncaptured turns since | **~57** (11-13 19:21 → 11-21 00:00): Pulse reports, delivery checks, morning briefs, `[System: …]` notices |
|
||||
| Writes from this session after that boundary | **0** |
|
||||
|
||||
- **The break sits in a 15-minute gap, and the 11-13 13:14 gateway restart sits inside it.** Last captured turn
|
||||
13:06:05, first uncaptured 13:21:00; the 11-13/11-14 audit record documents a **gateway restart at 13:14**
|
||||
that day (the one that cleared the stale `TELEGRAM_HOME_CHANNEL_THREAD_ID`). *Leading hypothesis:* the
|
||||
restart left this session's capture watermark at-or-past its tip, so the session syncs nothing while new
|
||||
sessions start clean. **NOT PROVEN** — the oldest surviving `agent.log` file begins **2026-11-13 19:03:34**,
|
||||
so the 13:14 window has rotated out of the log horizon and no startup banner is recoverable.
|
||||
- **Capture is not globally dead — it works for *new* sessions.** The single write after that boundary,
|
||||
`2026-11-18 11:25:03 UTC = 19:25:03 AWST`, came from `hermes_20261118_191808_5c9e08`, a session created by a
|
||||
genuine prompt that evening. That is the **only** write in eight days.
|
||||
- **The health monitor already says exactly this,** in its own words: `ea79a2107e36` is failing with
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`. The gate was right; the 11-20
|
||||
reading of it was wrong.
|
||||
|
||||
**Corrected statement for the record:**
|
||||
|
||||
> **Mnemosyne has captured nothing from the desktop DM session `20261031_190813_95fbde` since 2026-11-13
|
||||
> 13:10:35 AWST — ~8 days and ~57 user-role turns — while capturing normally from sessions created after that
|
||||
> date.**
|
||||
|
||||
**Counters (11-21 read):** `working_memory` **1,131** / `episodic_memory` **1,250** — both **unchanged** from
|
||||
the 11-20 read; global silence since the last write **52 h 35 m**. **Unclosed incidents: 4 → 5**
|
||||
(`cbfe43_fa12cca1ad4b` auth, `alerted_at: None`; `96eb95_98ade422d195` script, still alerting 11-21 00:31:16 (00:16:15 at the first read);
|
||||
`ea79a2_f2be2db3bfce` 11-19 12:01:58; `ea79a2_aae4d914f3c9` 11-20 00:03:36; **new** `ea79a2_ace0f263e18a`
|
||||
11-21 00:03:12). **Next probe:** replay one Pulse-shaped turn into that session with the provider's debug
|
||||
logging on, and diff against a pre-11-13 capture in `agent.log`.
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
---
|
||||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
title: Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||
---
|
||||
|
||||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||
# Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read
|
||||
|
||||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`** *(that is the **2026-11-20** reading — this page was written then; the re-read below carries the streak to **210**, `last_seen_at` **2026-11-21 00:31:16**)*.
|
||||
|
||||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||
re-reported every 15 minutes, not a series of separate faults.
|
||||
@@ -99,6 +99,22 @@ LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** th
|
||||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||
the fact pairs were not inspected.
|
||||
|
||||
## Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; **no signature fragmentation**
|
||||
|
||||
- `failure_streak` = **210** (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is
|
||||
11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red).
|
||||
- **Incident `96eb95_98ade422d195` is one row, not a series:** `first_seen_at` **2026-11-18 20:01:15**,
|
||||
`last_seen_at` **2026-11-21 00:31:16 (00:16:15 at the first read)**, `alerted_at` **2026-11-20 21:31:24**, `closed_at` empty. The 11-20
|
||||
page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication **holds**.
|
||||
- **The contrast is the health monitor**, which re-forms a **new incident id every slot** for one standing
|
||||
condition (`ea79a2_f2be2db3bfce` 11-19 12:01:58 → `ea79a2_aae4d914f3c9` 11-20 00:03:36 →
|
||||
`ea79a2_ace0f263e18a` 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when
|
||||
reading the ledger for "new" versus "chronic".
|
||||
- Error text unchanged: **59 unresolved conflicts against a threshold of 50**. Nothing was resolved,
|
||||
reclassified or deleted at this read either.
|
||||
- **Still the standing question:** a *level*-based test re-reports on every slot. A *state-change* alert would
|
||||
have fired once on 11-18 20:01 and then stayed quiet.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-28
|
||||
confidence: high
|
||||
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||
@@ -74,6 +74,31 @@ skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||
rather than at the alert throttle.
|
||||
|
||||
## Re-read 2026-11-21 00:0x AWST — **resolved**; and the ledger shows the outage was two nights, not one
|
||||
|
||||
- **The job is clean again.** `cbfe434a3b5e` reads `last_status: ok`, `failure_streak = 0`. The execution ledger
|
||||
dates the outage exactly:
|
||||
|
||||
| Slot | Execution | Outcome |
|
||||
|---|---|---|
|
||||
| 2026-11-18 00:00:08 | `d6323ba3b7ca4d8a95d0f57bf20b4807` | **failed** (`delivery_outcome: suppressed`) |
|
||||
| 2026-11-19 00:00:16 | `668984226ccc47b09cb66fe65ae89ddc` | **failed** (`delivery_outcome: suppressed`) |
|
||||
| 2026-11-20 00:00:08 | `b88bd7d8007248399f57cccfc98b536a` | **completed** (report not delivered) |
|
||||
| 2026-11-21 00:00:08 | `ec65ca362c6847fcb9405daa15d1f6ed` | running — the run authoring this note |
|
||||
|
||||
- **Two nights were lost, not one.** The 11-18 and 11-19 runs both failed, so windows **11-17 and 11-18** have
|
||||
no audit of their own. The **11-20 run absorbed both** via its three-window catch-up (2026-11-17 → 2026-11-20),
|
||||
read from the cron ledger and `state.db` rather than from conversation — `state.db` holds **no** interactive
|
||||
sessions in those windows, so there was no conversation to miss.
|
||||
- **No re-run of either lost slot was forced**; recovery came from the next scheduled slot running normally.
|
||||
- **Not re-tested:** whether the credit/auth fault can recur. The job's pin, provider and schedule are unchanged
|
||||
and no credential was rotated, so a recurrence would present identically.
|
||||
- **The silent-skip class this page identified is now observed, not inferred:** both failed nights carry
|
||||
`delivery_outcome: suppressed` and the incident row `cbfe43_fa12cca1ad4b` has `alerted_at: NULL` — two
|
||||
consecutive failed nights produced **no alert and no report**, which is why the outage only surfaced when the
|
||||
11-20 audit read the ledger. Whether that suppression is job-level config or framework behaviour is still
|
||||
**UNVERIFIED** (the job's delivery config was not read).
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: Two digests died on `content_policy_blocked` (2026-11-20) — first content-filter failures on record
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [cron, content-policy, provider-error, gmail, freshrss, digests, incident, unclassified-failure]
|
||||
sources: [live-probe-cron-executions-db-2026-11-21, live-probe-cron-incidents-2026-11-21, live-probe-jobs-json-2026-11-21]
|
||||
---
|
||||
|
||||
# Two digests died on `content_policy_blocked` (2026-11-20)
|
||||
|
||||
**The first content-filter failures in the retained log.** Two unrelated jobs, on two different feeds, died
|
||||
with the *identical* terminal string on 2026-11-20 — hours apart, with every other job in the window clean.
|
||||
|
||||
## What happened
|
||||
|
||||
| Job | Job ID | Slot (AWST) | Execution | Duration | Terminal error |
|
||||
|---|---|---|---|---|---|
|
||||
| Gmail Action Digest | `08eb86f02d37` | 08:31:06 → 08:32:45 | `a490b74a509e4303acad671ee1ee80cf` | 1m39s | `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | 12:01:14 → 12:05:04 | `0e1c11f49313428799de604afe73595c` | 3m50s | same string |
|
||||
|
||||
Both jobs are pinned `provider: custom:omni-101` / `model: smart-route`. Both carry
|
||||
**`failure_streak = 1`** — i.e. **isolated single failures, not chronic**: the preceding slot of each job
|
||||
completed. FreshRSS Midday ran clean on 11-19 12:01.
|
||||
|
||||
## What is established
|
||||
|
||||
- The failure is raised **by the model-facing API, not by the job logic**. The string is a provider content
|
||||
filter surfaced to the job as a wrapped `RuntimeError`; the digest scripts assembled their input fine.
|
||||
- **Both jobs alerted, and both alerts are unclosed.** Incident rows:
|
||||
- `08eb86_c0cca1a55b96` — first seen 11-20 08:32:44, `alerted_at` 08:32:46
|
||||
- `ecc1fc_5a10b70830d1` — first seen 11-20 12:05:03, `alerted_at` 12:05:05
|
||||
|
||||
This is the **opposite** of [[incidents/2026-11-18-wiki-audit-credit-outage]], which failed silently for two
|
||||
nights: here the alerting path worked end to end.
|
||||
- `failure_type` is recorded as **`unknown`** for both. The incident classifier does not recognise this
|
||||
class, so it can neither group nor auto-close it — a recurrence will alert, but will not be understood.
|
||||
- Both left a full transcript: `~/.hermes/cron/output/08eb86f02d37/2026-11-20_08-32-44.md` and
|
||||
`~/.hermes/cron/output/ecc1fc13655a/2026-11-20_12-05-03.md`.
|
||||
|
||||
## What is NOT established
|
||||
|
||||
- **Whether the trigger was the request or the response.** The message reads "Output data may contain
|
||||
inappropriate content", but a filter that scans a request payload reports the same way. The Gmail digest
|
||||
reads mail; the FreshRSS digest reads news — the shared factor is more likely a *token* the router's filter
|
||||
dislikes than the subject matter. **The two output files above settle this**: diff what the two prompts had
|
||||
in common.
|
||||
- **Whether the router (`omni-101`) applies a filter its upstreams do not.** If it does, the same feeds can
|
||||
fail again with nothing changed on this host. Probe: replay one of the two prompts against the router and
|
||||
against an upstream provider directly, and compare.
|
||||
- Whether it recurs. Two slots, two jobs, one day is a pattern worth watching — not yet a trend.
|
||||
|
||||
## Standing risk
|
||||
|
||||
Two of the estate's daily digests sit behind a provider filter that has already rejected them once, and the
|
||||
classifier files the failure as `unknown` — so a recurrence will alert without being recognised. Cheapest
|
||||
mitigation if it recurs: **no content-policy string is worth a lost digest** — route the two digest jobs to a
|
||||
provider without the filter, or strip the highest-risk section from the prompt payload.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[systems/mnemosyne]]
|
||||
Reference in New Issue
Block a user