wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)
This commit is contained in:
@@ -202,3 +202,30 @@ Primary agent memory store for Hermes.
|
||||
rather than fail. The **underlying cold start was not optimised** — only the budget was raised.
|
||||
- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic
|
||||
1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.
|
||||
|
||||
## Re-read 2026-11-21 (audit window 2026-11-20; live probes 2026-11-21 00:0x–00:5x AWST)
|
||||
|
||||
**The capture fault is *session-scoped*, and its boundary is exact — the 11-20 "conversational silence"
|
||||
reading is withdrawn.**
|
||||
|
||||
- **One session has stopped capturing entirely.** The desktop DM session `20261031_190813_95fbde` owns
|
||||
**124 Pulse/automation rows** in `working_memory` (84 under the bare id, 40 under
|
||||
`hermes_agent:main:local:dm:`), spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (= 13:10:35 AWST)**
|
||||
and **nothing after** — while the session itself kept producing user-role turns. Verified against
|
||||
`state.db`: last captured turn **13:06:05 AWST**, first uncaptured **13:21:00 AWST**, **~57 turns** since.
|
||||
The **11-13 13:14 gateway restart** sits inside that 15-minute boundary and is the leading hypothesis (a
|
||||
watermark left at-or-past the tip). **NOT PROVEN** — the oldest surviving `agent.log` begins
|
||||
**2026-11-13 19:03:34**, so the window has rotated out of the log horizon.
|
||||
- **Capture still works for *new* sessions.** The single write after that boundary —
|
||||
`2026-11-18 11:25:03 UTC = 19:25:03 AWST` — came from `hermes_20261118_191808_5c9e08`, a session created by a
|
||||
genuine prompt that evening. The pipeline is not globally dead; it fails for the long-lived session.
|
||||
- **Global silence now 52 h 35 m** (11-18 19:25:03 AWST → this read); exactly **one** write in eight days.
|
||||
`working_memory` **1,131** / `episodic_memory` **1,250** — both unchanged since the 11-20 read.
|
||||
- **The health monitor was already saying this.** `ea79a2107e36` fails with
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`; incident
|
||||
`ea79a2_ace0f263e18a` first seen 2026-11-21 00:03:12, alerted 00:03:17, unclosed.
|
||||
- **Consolidation still 0:** 199 of the last 200 cycles consolidated nothing.
|
||||
- **Conflicts still 59** against a threshold of 50; the watchdog has now been red for **210 consecutive
|
||||
15-minute slots** (11-18 20:01:15 → 11-21 00:31:16 (00:16:15 at the first read)).
|
||||
|
||||
→ [[incidents/2026-11-14-mnemosyne-capture-stall]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||
|
||||
@@ -1360,3 +1360,55 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
|
||||
condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both
|
||||
- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level.
|
||||
|
||||
## Re-verified 2026-11-21 (window 2026-11-20, AWST; live reads 2026-11-21 00:0x–00:5x)
|
||||
|
||||
**Fleet flat at 29 jobs / 26 enabled** (twelfth-plus consecutive window). Ledger, deliveries and the two
|
||||
standing incidents re-read; **two new failure signatures appeared on 11-20**, both raised by the model-facing
|
||||
API rather than by job logic.
|
||||
|
||||
### Non-clean `last_status`
|
||||
|
||||
| Job | ID | Schedule | Streak | Cause |
|
||||
|---|---|---|---|---|
|
||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | **210** | 59 unresolved conflicts vs threshold 50 (standing since 11-18 20:01) |
|
||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | **4** | `STALE MEMORY PIPELINE: capture missed: user message` |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | **1** | `content_policy_blocked` (11-20 08:31:06) |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | **1** | `content_policy_blocked` (11-20 12:01:14) |
|
||||
|
||||
All four are `enabled: true`. `failure_streak = 1` on the two digests means **isolated failures, not chronic** —
|
||||
the preceding slot of each completed (FreshRSS Midday ran clean on 11-19 12:01).
|
||||
|
||||
### New failure class: provider content filter
|
||||
|
||||
Both digests died on the identical string — `RuntimeError: content_policy_blocked: Output data may contain
|
||||
inappropriate content.` — hours apart, with every other job in the window clean. Both **alerted** (incident
|
||||
rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`, both unclosed) and both are classified
|
||||
**`failure_type: unknown`**, so the classifier cannot group or auto-close this class.
|
||||
→ [[incidents/2026-11-20-content-policy-blocked-digest-failures]]
|
||||
|
||||
### Standing reds
|
||||
|
||||
- **Watchdog: 210 consecutive 15-minute slots**, one stable incident id `96eb95_98ade422d195`
|
||||
(`first_seen_at` 11-18 20:01:15, `last_seen_at` 11-21 00:31:16 (00:16:15 at the first read), `alerted_at` 11-20 21:31:24, unclosed) —
|
||||
**no signature fragmentation**, contrary to the 11-20 page's concern.
|
||||
- **Health monitor: the opposite.** One standing condition re-forms a **fresh incident id every slot**
|
||||
(`ea79a2_f2be2db3bfce` 11-19 12:01:58, `ea79a2_aae4d914f3c9` 11-20 00:03:36, `ea79a2_ace0f263e18a`
|
||||
11-21 00:03:12) — three rows for one fault. The two monitors differ in de-duplication behaviour and it is
|
||||
worth knowing which is which when reading the ledger.
|
||||
- **Capture:** `working_memory` **1,131** / `episodic_memory` **1,250**, unchanged; corrected diagnosis on
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||
- **Consolidation 0 items** in 199 of the last 200 cycles.
|
||||
|
||||
### Freshness / files touched
|
||||
|
||||
- `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` →
|
||||
**5 stale / 131 fresh / 3 no-date of 139 pages**. The five at exactly 21 d are the predicted cohort:
|
||||
`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`.
|
||||
- **Pages touched by this audit:** [[incidents/2026-11-20-content-policy-blocked-digest-failures]] (new),
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] (title + fifth read),
|
||||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (115 → 210 slots),
|
||||
[[incidents/2026-11-18-wiki-audit-credit-outage]] (→ resolved), [[systems/mnemosyne]],
|
||||
[[current-state]], [[index]], `log.md`.
|
||||
- **No job added, retired, disabled or repinned; no page removed or archived; no system obsoleted; no decision
|
||||
reversed.**
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user