wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)
This commit is contained in:
+16
-5
@@ -3,16 +3,27 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db, live-sqlite-mnemosyne]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-11-20
|
||||
Updated: 2026-11-21
|
||||
|
||||
## Recent Changes (2026-11-20, nightly audit; live probes 2026-11-21 00:0x – 00:5x AWST)
|
||||
|
||||
**Three things moved: the capture fault was re-diagnosed as session-scoped (correcting the 11-20 read), two digests died on a provider content filter, and the audit itself came back.**
|
||||
|
||||
- **⚠ Capture — the 11-20 "every interactive session was captured" finding is WITHDRAWN.** That read counted *sessions*; capture is per *message*. Per message, the desktop DM session `20261031_190813_95fbde` owns **124** Pulse/automation rows in `working_memory`, spanning **2026-10-31 11:09:16 → 2026-11-13 13:10:35 AWST**, and **nothing after** — while the session kept producing turns: **~57** since (last captured turn **13:06:05**, first uncaptured **13:21:00**, verified in `state.db`). The **11-13 13:14 gateway restart** sits inside that 15-minute boundary — **leading hypothesis, explicitly NOT proven** (`agent.log`'s oldest surviving file begins **11-13 19:03:34**; the window has rotated out). Capture is **not** globally dead: the only write since, **`2026-11-18 19:25:03` AWST**, came from the new session `hermes_20261118_191808_5c9e08`. Counters frozen at `working_memory` **1,131** / `episodic_memory` **1,250**; silence since the last write **52 h 35 m**. The health monitor already reports it in its own words: `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak: 4`. → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
|
||||
- **New failure class — `content_policy_blocked`.** Two unrelated digests died on the identical provider string on 11-20: **Gmail Action Digest** `08eb86f02d37` at **08:31:06** and **FreshRSS Midday News Briefing** `ecc1fc13655a` at **12:01:14**. Both `failure_streak: 1` (**isolated**), both **alerted**, both **unclosed**, both classified `failure_type: unknown`. → [[incidents/2026-11-20-content-policy-blocked-digest-failures]]
|
||||
- **The audit came back — and the outage was two nights, not one.** `cbfe434a3b5e` reads `last_status: ok` / streak 0. Ledger: **11-18 00:00 failed**, **11-19 00:00 failed**, **11-20 00:00 completed** (report not delivered — `delivery_outcome: suppressed`), **11-21 00:00** this run. The 11-20 run absorbed windows **11-17 and 11-18** via its three-window catch-up. **No re-run was forced.** → [[incidents/2026-11-18-wiki-audit-credit-outage]] now `resolved`.
|
||||
- **Watchdog still red — 210 consecutive 15-minute slots** (was 115 at the 11-20 read), **59 unresolved conflicts vs a threshold of 50**. Its incident `96eb95_98ade422d195` held **one id** from **11-18 20:01:15** to **11-21 00:31:16 (00:16:15 at the first read)** (**no fragmentation**) — unlike the **health monitor**, which re-forms a fresh incident id every slot for one standing condition (`ea79a2_f2be2db3bfce` → `…aae4d914f3c9` → `…ace0f263e18a`). → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||
- **Consolidation still zero-item:** 199 of the last 200 cycles. Fleet **29 jobs / 26 enabled** (flat). Freshness audit **5 stale / 131 fresh / 3 no-date of 139** — the five at exactly 21 d are the predicted cohort (`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`).
|
||||
- **Pages touched:** [[incidents/2026-11-20-content-policy-blocked-digest-failures]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (title + fifth read), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (115 → 210), [[incidents/2026-11-18-wiki-audit-credit-outage]] (→ resolved), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-21**.
|
||||
|
||||
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
title: Mnemosyne capture is dead for the long-running desktop DM session since 2026-11-13 13:10 AWST — ~57 uncaptured turns, four unclosed health-monitor alerts
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, session-scoped, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
@@ -200,6 +200,47 @@ silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98a
|
||||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||
|
||||
### Fifth read — 2026-11-21 00:0x–00:5x AWST: **the 11-20 conclusion above is withdrawn.** Capture is not "quiet", it is **dead for one specific session**, and the boundary is exact.
|
||||
|
||||
The 11-20 read counted **sessions** in `state.db`, found only three in the window, and concluded that every
|
||||
interactive session had been captured — so the alert was measuring conversational silence. **The wrong unit.**
|
||||
Capture is per *message*. Counted per message, the picture inverts:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Rows this session owns in `working_memory` | **124** (84 under the bare id `hermes_20261031_190813_95fbde`, 40 under `hermes_agent:main:local:dm:20261031_190813_95fbde`) |
|
||||
| Their span | **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC = 2026-11-13 13:10:35 AWST** — then nothing |
|
||||
| Last *captured* turn in `state.db` | **2026-11-13 13:06:05 AWST** (Pulse report `pc36e5568`, row written 13:10:35) |
|
||||
| First *uncaptured* turn | **2026-11-13 13:21:00 AWST** (`Pulse delivery check p65be4717`) — and every turn since |
|
||||
| Uncaptured turns since | **~57** (11-13 19:21 → 11-21 00:00): Pulse reports, delivery checks, morning briefs, `[System: …]` notices |
|
||||
| Writes from this session after that boundary | **0** |
|
||||
|
||||
- **The break sits in a 15-minute gap, and the 11-13 13:14 gateway restart sits inside it.** Last captured turn
|
||||
13:06:05, first uncaptured 13:21:00; the 11-13/11-14 audit record documents a **gateway restart at 13:14**
|
||||
that day (the one that cleared the stale `TELEGRAM_HOME_CHANNEL_THREAD_ID`). *Leading hypothesis:* the
|
||||
restart left this session's capture watermark at-or-past its tip, so the session syncs nothing while new
|
||||
sessions start clean. **NOT PROVEN** — the oldest surviving `agent.log` file begins **2026-11-13 19:03:34**,
|
||||
so the 13:14 window has rotated out of the log horizon and no startup banner is recoverable.
|
||||
- **Capture is not globally dead — it works for *new* sessions.** The single write after that boundary,
|
||||
`2026-11-18 11:25:03 UTC = 19:25:03 AWST`, came from `hermes_20261118_191808_5c9e08`, a session created by a
|
||||
genuine prompt that evening. That is the **only** write in eight days.
|
||||
- **The health monitor already says exactly this,** in its own words: `ea79a2107e36` is failing with
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`. The gate was right; the 11-20
|
||||
reading of it was wrong.
|
||||
|
||||
**Corrected statement for the record:**
|
||||
|
||||
> **Mnemosyne has captured nothing from the desktop DM session `20261031_190813_95fbde` since 2026-11-13
|
||||
> 13:10:35 AWST — ~8 days and ~57 user-role turns — while capturing normally from sessions created after that
|
||||
> date.**
|
||||
|
||||
**Counters (11-21 read):** `working_memory` **1,131** / `episodic_memory` **1,250** — both **unchanged** from
|
||||
the 11-20 read; global silence since the last write **52 h 35 m**. **Unclosed incidents: 4 → 5**
|
||||
(`cbfe43_fa12cca1ad4b` auth, `alerted_at: None`; `96eb95_98ade422d195` script, still alerting 11-21 00:31:16 (00:16:15 at the first read);
|
||||
`ea79a2_f2be2db3bfce` 11-19 12:01:58; `ea79a2_aae4d914f3c9` 11-20 00:03:36; **new** `ea79a2_ace0f263e18a`
|
||||
11-21 00:03:12). **Next probe:** replay one Pulse-shaped turn into that session with the provider's debug
|
||||
logging on, and diff against a pre-11-13 capture in `agent.log`.
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
---
|
||||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
title: Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||
---
|
||||
|
||||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||
# Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read
|
||||
|
||||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`** *(that is the **2026-11-20** reading — this page was written then; the re-read below carries the streak to **210**, `last_seen_at` **2026-11-21 00:31:16**)*.
|
||||
|
||||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||
re-reported every 15 minutes, not a series of separate faults.
|
||||
@@ -99,6 +99,22 @@ LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** th
|
||||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||
the fact pairs were not inspected.
|
||||
|
||||
## Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; **no signature fragmentation**
|
||||
|
||||
- `failure_streak` = **210** (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is
|
||||
11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red).
|
||||
- **Incident `96eb95_98ade422d195` is one row, not a series:** `first_seen_at` **2026-11-18 20:01:15**,
|
||||
`last_seen_at` **2026-11-21 00:31:16 (00:16:15 at the first read)**, `alerted_at` **2026-11-20 21:31:24**, `closed_at` empty. The 11-20
|
||||
page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication **holds**.
|
||||
- **The contrast is the health monitor**, which re-forms a **new incident id every slot** for one standing
|
||||
condition (`ea79a2_f2be2db3bfce` 11-19 12:01:58 → `ea79a2_aae4d914f3c9` 11-20 00:03:36 →
|
||||
`ea79a2_ace0f263e18a` 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when
|
||||
reading the ledger for "new" versus "chronic".
|
||||
- Error text unchanged: **59 unresolved conflicts against a threshold of 50**. Nothing was resolved,
|
||||
reclassified or deleted at this read either.
|
||||
- **Still the standing question:** a *level*-based test re-reports on every slot. A *state-change* alert would
|
||||
have fired once on 11-18 20:01 and then stayed quiet.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-28
|
||||
confidence: high
|
||||
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||
@@ -74,6 +74,31 @@ skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||
rather than at the alert throttle.
|
||||
|
||||
## Re-read 2026-11-21 00:0x AWST — **resolved**; and the ledger shows the outage was two nights, not one
|
||||
|
||||
- **The job is clean again.** `cbfe434a3b5e` reads `last_status: ok`, `failure_streak = 0`. The execution ledger
|
||||
dates the outage exactly:
|
||||
|
||||
| Slot | Execution | Outcome |
|
||||
|---|---|---|
|
||||
| 2026-11-18 00:00:08 | `d6323ba3b7ca4d8a95d0f57bf20b4807` | **failed** (`delivery_outcome: suppressed`) |
|
||||
| 2026-11-19 00:00:16 | `668984226ccc47b09cb66fe65ae89ddc` | **failed** (`delivery_outcome: suppressed`) |
|
||||
| 2026-11-20 00:00:08 | `b88bd7d8007248399f57cccfc98b536a` | **completed** (report not delivered) |
|
||||
| 2026-11-21 00:00:08 | `ec65ca362c6847fcb9405daa15d1f6ed` | running — the run authoring this note |
|
||||
|
||||
- **Two nights were lost, not one.** The 11-18 and 11-19 runs both failed, so windows **11-17 and 11-18** have
|
||||
no audit of their own. The **11-20 run absorbed both** via its three-window catch-up (2026-11-17 → 2026-11-20),
|
||||
read from the cron ledger and `state.db` rather than from conversation — `state.db` holds **no** interactive
|
||||
sessions in those windows, so there was no conversation to miss.
|
||||
- **No re-run of either lost slot was forced**; recovery came from the next scheduled slot running normally.
|
||||
- **Not re-tested:** whether the credit/auth fault can recur. The job's pin, provider and schedule are unchanged
|
||||
and no credential was rotated, so a recurrence would present identically.
|
||||
- **The silent-skip class this page identified is now observed, not inferred:** both failed nights carry
|
||||
`delivery_outcome: suppressed` and the incident row `cbfe43_fa12cca1ad4b` has `alerted_at: NULL` — two
|
||||
consecutive failed nights produced **no alert and no report**, which is why the outage only surfaced when the
|
||||
11-20 audit read the ledger. Whether that suppression is job-level config or framework behaviour is still
|
||||
**UNVERIFIED** (the job's delivery config was not read).
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: Two digests died on `content_policy_blocked` (2026-11-20) — first content-filter failures on record
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-21
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
review_after: 2026-11-22
|
||||
confidence: high
|
||||
tags: [cron, content-policy, provider-error, gmail, freshrss, digests, incident, unclassified-failure]
|
||||
sources: [live-probe-cron-executions-db-2026-11-21, live-probe-cron-incidents-2026-11-21, live-probe-jobs-json-2026-11-21]
|
||||
---
|
||||
|
||||
# Two digests died on `content_policy_blocked` (2026-11-20)
|
||||
|
||||
**The first content-filter failures in the retained log.** Two unrelated jobs, on two different feeds, died
|
||||
with the *identical* terminal string on 2026-11-20 — hours apart, with every other job in the window clean.
|
||||
|
||||
## What happened
|
||||
|
||||
| Job | Job ID | Slot (AWST) | Execution | Duration | Terminal error |
|
||||
|---|---|---|---|---|---|
|
||||
| Gmail Action Digest | `08eb86f02d37` | 08:31:06 → 08:32:45 | `a490b74a509e4303acad671ee1ee80cf` | 1m39s | `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | 12:01:14 → 12:05:04 | `0e1c11f49313428799de604afe73595c` | 3m50s | same string |
|
||||
|
||||
Both jobs are pinned `provider: custom:omni-101` / `model: smart-route`. Both carry
|
||||
**`failure_streak = 1`** — i.e. **isolated single failures, not chronic**: the preceding slot of each job
|
||||
completed. FreshRSS Midday ran clean on 11-19 12:01.
|
||||
|
||||
## What is established
|
||||
|
||||
- The failure is raised **by the model-facing API, not by the job logic**. The string is a provider content
|
||||
filter surfaced to the job as a wrapped `RuntimeError`; the digest scripts assembled their input fine.
|
||||
- **Both jobs alerted, and both alerts are unclosed.** Incident rows:
|
||||
- `08eb86_c0cca1a55b96` — first seen 11-20 08:32:44, `alerted_at` 08:32:46
|
||||
- `ecc1fc_5a10b70830d1` — first seen 11-20 12:05:03, `alerted_at` 12:05:05
|
||||
|
||||
This is the **opposite** of [[incidents/2026-11-18-wiki-audit-credit-outage]], which failed silently for two
|
||||
nights: here the alerting path worked end to end.
|
||||
- `failure_type` is recorded as **`unknown`** for both. The incident classifier does not recognise this
|
||||
class, so it can neither group nor auto-close it — a recurrence will alert, but will not be understood.
|
||||
- Both left a full transcript: `~/.hermes/cron/output/08eb86f02d37/2026-11-20_08-32-44.md` and
|
||||
`~/.hermes/cron/output/ecc1fc13655a/2026-11-20_12-05-03.md`.
|
||||
|
||||
## What is NOT established
|
||||
|
||||
- **Whether the trigger was the request or the response.** The message reads "Output data may contain
|
||||
inappropriate content", but a filter that scans a request payload reports the same way. The Gmail digest
|
||||
reads mail; the FreshRSS digest reads news — the shared factor is more likely a *token* the router's filter
|
||||
dislikes than the subject matter. **The two output files above settle this**: diff what the two prompts had
|
||||
in common.
|
||||
- **Whether the router (`omni-101`) applies a filter its upstreams do not.** If it does, the same feeds can
|
||||
fail again with nothing changed on this host. Probe: replay one of the two prompts against the router and
|
||||
against an upstream provider directly, and compare.
|
||||
- Whether it recurs. Two slots, two jobs, one day is a pattern worth watching — not yet a trend.
|
||||
|
||||
## Standing risk
|
||||
|
||||
Two of the estate's daily digests sit behind a provider filter that has already rejected them once, and the
|
||||
classifier files the failure as `unknown` — so a recurrence will alert without being recognised. Cheapest
|
||||
mitigation if it recurs: **no content-policy string is worth a lost digest** — route the two digest jobs to a
|
||||
provider without the filter, or strip the highest-risk section from the prompt payload.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[systems/mnemosyne]]
|
||||
@@ -3,16 +3,21 @@ title: Anthony's Systems Wiki
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout, content-policy, session-scoped-capture]
|
||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, live-sqlite-mnemosyne, raw/stocktake]
|
||||
---
|
||||
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. last updated: 2026-11-20
|
||||
> model routing and automation. last updated: 2026-11-21
|
||||
>
|
||||
> **2026-11-21 (nightly audit; window 2026-11-20; live probes 2026-11-21 00:0x – 00:5x AWST) — the memory fault is session-scoped, a new provider content filter killed two digests, and the audit came back.**
|
||||
> **⚠ Capture, corrected a second time.** The 11-20 finding that "**every** interactive session was captured" is **withdrawn** — that read counted *sessions*; capture is per *message*. Per message: the desktop DM session `20261031_190813_95fbde` owns **124** Pulse/automation rows in `working_memory` from **2026-10-31 11:09:16** to **2026-11-13 13:10:35 AWST** and **nothing after**, while the session kept producing turns — **~57** since, last captured turn **13:06:05**, first uncaptured **13:21:00** (verified in `state.db`). The **11-13 13:14 gateway restart** sits inside that 15-minute boundary — **leading hypothesis, explicitly NOT proven** (`agent.log`'s oldest surviving file begins **11-13 19:03:34**; the window has rotated out). Capture is **not** globally dead: the only write since, **`2026-11-18 19:25:03` AWST**, came from the new session `hermes_20261118_191808_5c9e08`. Counters frozen at `working_memory` **1,131** / `episodic_memory` **1,250**; **52 h 35 m** since the last write; the health monitor already says it — `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak: 4` → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]].
|
||||
> **New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string on 11-20 — **Gmail Action Digest `08eb86f02d37` 08:31:06** and **FreshRSS Midday News Briefing `ecc1fc13655a` 12:01:14** — `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both `failure_streak: 1` (**isolated**, not chronic), both **alerted** and **unclosed**, both `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]].
|
||||
> **The audit came back — and the outage was two nights, not one.** `cbfe434a3b5e` is `ok` / streak 0; the ledger dates it **11-18 00:00 failed · 11-19 00:00 failed · 11-20 00:00 completed** (report not delivered — `delivery_outcome: suppressed`) **· 11-21 00:00** this run. The 11-20 run absorbed windows **11-17/11-18** via its three-window catch-up; **no re-run was forced** → [[incidents/2026-11-18-wiki-audit-credit-outage]] now **`resolved`**. The **Mnemosyne Watchdog is still red — 210 consecutive 15-minute slots** (59 conflicts vs threshold 50), holding **one** incident id `96eb95_98ade422d195` from **11-18 20:01:15** to **11-21 00:31:16 (00:16:15 at the first read)** (**no fragmentation**), whereas the **health monitor** re-forms a fresh incident id every slot for one standing condition → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. Consolidation **0 items** (199 of the last 200 cycles). Fleet **29 jobs / 26 enabled**. Freshness **5 stale / 131 fresh / 3 no-date** of 139.
|
||||
>
|
||||
> **2026-11-20 (three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) — the memory freeze broke by itself, the alarm did not, and the audits that would have said so had died twice.**
|
||||
> **The capture stall is over — silently.** `working_memory` **1130 → 1131** / `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST**: **105 h 52 m**, ended with **no intervention**. And the "capture is dead" reading was wrong — `state.db` holds only **three** non-cron sessions in that window (two `desktop` on 11-14, both *before* the 09:33 boundary, then the 11-18 **19:18** session whose turn is the captured row), so **every interactive session was captured**; the alert measures **conversational silence** (age of the newest non-`cron`/`subagent` message). Corrected on [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]].
|
||||
@@ -93,11 +98,12 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
|
||||
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **OPEN — the nightly audit failed on two consecutive nights, so 2026-11-17 and 2026-11-18 were never audited, and no failure notice was ever sent.** `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`** — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. The 2026-11-20 00:00 slot (this run) covers all three windows.
|
||||
- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 115 consecutive watchdog failures from one cause: the conflict queue.** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 115`** at **2026-11-20 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 115 faults. Each slot delivers a *failure notice* (9 of the 24 deliveries since 11-18 are `for_failure: 1`), so the noisy channel is also the only channel; every slot re-opens incident `96eb95_98ade422d195` and nothing escalates or closes.
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (corrected 2026-11-20) — the freeze ended by itself at 2026-11-18 19:25 AWST after 105 h 52 m, and the "captured nothing" reading was a quiet estate, not a broken hook.** Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
|
||||
- [[incidents/2026-11-20-content-policy-blocked-digest-failures]] - **OPEN — first `content_policy_blocked` failures on record: two unrelated digests killed by the identical provider string hours apart.** **Gmail Action Digest** `08eb86f02d37` ran **2026-11-20 08:31:06 → 08:32:45** and **FreshRSS Midday News Briefing** `ecc1fc13655a` ran from **12:01:14**, both ending `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both carry **`failure_streak: 1`** (isolated — each job's preceding slot completed), both **alerted** (incident rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`) and both still **unclosed**, and both are classified **`failure_type: unknown`**, so the classifier cannot group the class or auto-close it. Nothing was changed on either job and **no re-run was forced**; whether the class recurs is unknown.
|
||||
- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **RESOLVED 2026-11-21 — the nightly audit failed on two consecutive nights, so windows 2026-11-17 and 2026-11-18 were never audited on their own night, and no failure notice was ever sent.** The ledger dates it exactly: `cbfe434a3b5e` failed **2026-11-18T00:00:08** and **2026-11-19T00:00:16** on a chained-provider **credit/auth** fault, then **completed on 11-20T00:00:08** (report not delivered — `delivery_outcome: suppressed`), absorbing both lost windows via its three-window catch-up; the job now reads `last_status: ok` / `failure_streak: 0`. **No re-run was forced**, and the silent-skip class it identified stays open on [[systems/scheduled-tasks]]. `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) was **never alerted** (`alerted_at: None`) — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] — and is now **resolved** (`closed_at` **2026-11-20 01:19:13**, i.e. it cleared itself on the first clean completion).
|
||||
- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 210 consecutive watchdog failures from one cause: the conflict queue (115 at the 11-20 read).** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 210`** at **2026-11-21 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 210 faults. Each slot delivers a *failure notice*, so the noisy channel is also the only channel. Unlike the health monitor, this job **de-duplicates**: **one** incident row `96eb95_98ade422d195` (`failure_type: script`) has carried all 210 slots — `first_seen` 2026-11-18 20:01:15, `last_seen` **2026-11-21 00:31:16**, `alerted_at` 2026-11-20 21:31:24, still **unclosed** — so nothing escalates and nothing closes.
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (re-corrected 2026-11-21) — the 11-20 "quiet estate, not a broken hook" reading is WITHDRAWN.** Counted per *message* rather than per *session*, the desktop DM session `20261031_190813_95fbde` owns **124** `working_memory` rows ending **2026-11-13 13:10:35 AWST** with **~57** of its turns uncaptured since (last captured turn 13:06:05, first uncaptured 13:21:00; the **11-13 13:14 gateway restart** sits inside that gap — leading hypothesis, not proven). The **105 h 52 m** global freeze did end by itself at **2026-11-18 19:25 AWST** and capture still works for **new** sessions; the failing one is the long-lived desktop session. Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
|
||||
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
|
||||
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
|
||||
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion — its incident row was closed 2026-11-12T00:25:36; *cron_incidents* stood at 137 rows at that read) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and was **never alerted** (`alerted_at: NULL`) and is now **resolved** (`closed_at` **2026-11-12T00:25:36**) — so "the estate's only open incident row" has been overtaken: unclosed *cron_incidents* at the **2026-11-21** read is **7** (`08eb86` ×1, `ecc1fc` ×1, `ea79a2` ×4, `96eb95` ×1). Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
|
||||
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
|
||||
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.
|
||||
|
||||
@@ -1890,3 +1890,61 @@ recorded here so the next audit does not re-litigate it.
|
||||
- `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27).
|
||||
- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists).
|
||||
- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced.
|
||||
|
||||
---
|
||||
|
||||
## 2026-11-21 00:0x–00:5x AWST — nightly wiki audit (window: 2026-11-20)
|
||||
|
||||
**Headline: the memory capture fault is *session-scoped*, and its boundary is exact — the 11-20
|
||||
"conversational silence" conclusion is withdrawn.**
|
||||
|
||||
The desktop DM session `20261031_190813_95fbde` owns **124 Pulse/automation rows** in `working_memory`,
|
||||
spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (13:10:35 AWST)** and **nothing after** — while the
|
||||
session itself kept producing turns: **~57** since, last captured turn **13:06:05 AWST**, first uncaptured
|
||||
**13:21:00 AWST** (verified against `state.db`). The **11-13 13:14 gateway restart** sits inside that
|
||||
15-minute boundary; that is the **leading hypothesis, explicitly NOT proven** — the oldest surviving
|
||||
`agent.log` file begins **11-13 19:03:34**, so the window has rotated out of the log horizon. Capture is
|
||||
**not** globally dead: the single write since, `2026-11-18 19:25:03 AWST`, came from the new session
|
||||
`hermes_20261118_191808_5c9e08`. Global silence now **52 h 35 m**; counters frozen at `working_memory`
|
||||
**1,131** / `episodic_memory` **1,250**. The health monitor is already reporting this in its own words —
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`.
|
||||
|
||||
**New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string
|
||||
on 11-20 — Gmail Action Digest **08:31:06** (`08eb86f02d37`) and FreshRSS Midday News Briefing **12:01:14**
|
||||
(`ecc1fc13655a`), both `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.`
|
||||
Both carry `failure_streak = 1` (**isolated**, not chronic), both **alerted** and both **unclosed**, both
|
||||
classified `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]].
|
||||
|
||||
**Watchdog: 210 consecutive 15-minute slots** (was 115 at the 11-20 read). Incident
|
||||
`96eb95_98ade422d195` held **one id** from 11-18 20:01:15 to 11-21 00:31:16 (00:16:15 at the first read) — **no fragmentation**, unlike the
|
||||
health monitor, which re-forms a fresh incident id every slot for a single standing condition. 59 conflicts
|
||||
against a threshold of 50. Consolidation still **0 items** (199 of the last 200 cycles).
|
||||
|
||||
**The wiki-audit outage is resolved.** `cbfe434a3b5e` now reads `last_status: ok` / `failure_streak: 0`, and
|
||||
the execution ledger dates the outage exactly: **11-18 00:00:08 failed**, **11-19 00:00:16 failed**, **11-20
|
||||
00:00:08 completed** (report not delivered — `delivery_outcome: suppressed`), and **11-21 00:00:08** is this
|
||||
run. So **two** nights were lost (windows 11-17 and 11-18); the 11-20 run absorbed both via its three-window
|
||||
catch-up, which read the cron ledger and `state.db` rather than conversation. **No re-run was forced.**
|
||||
→ [[incidents/2026-11-18-wiki-audit-credit-outage]] now `resolved`.
|
||||
|
||||
**Ledger confirmation read at the end of this run** (`~/.hermes/cron/executions.db`, `cron_incidents`, 2026-11-21 00:3x):
|
||||
its incident row `cbfe43_fa12cca1ad4b` (`auth`) was **never alerted** and is **closed** (`closed_at` **2026-11-20 01:19:13**) — it
|
||||
cleared itself on the first clean completion. Unclosed rows stand at **7**, and none of them belong to the audit:
|
||||
`08eb86_c0cca1a55b96` + `ecc1fc_5a10b70830d1` (the two `content_policy_blocked` digests, `unknown`),
|
||||
four `ea79a2_*` health-monitor rows (`auth`, one per slot: `f2be2db3bfce`, `aae4d914f3c9`, `51c605d08ea5`, `ace0f263e18a`),
|
||||
and `96eb95_98ade422d195` (`script`, the conflict-queue trip). The same read moved the watchdog's `last_seen_at`
|
||||
to **00:31:16** — the streak is still live, so every "hold one id" timestamp on these pages is a moving target;
|
||||
the pages now cite **00:31:16 (00:16:15 at the first read)**. Earlier standing claims corrected in the same pass:
|
||||
the health monitor's unclosed count is **four** (not five) — the capture-stall title now says so — and
|
||||
[[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]'s "estate's only open incident row" is overtaken.
|
||||
|
||||
**Freshness:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` →
|
||||
**5 stale / 131 fresh / 3 no-date of 139 pages**; the five at exactly 21 d are the predicted cohort
|
||||
(`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`).
|
||||
|
||||
**No job added, retired, disabled or repinned; no wiki page removed or archived; no decision reversed.**
|
||||
Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.md`;
|
||||
`incidents/2026-11-14-mnemosyne-capture-stall.md` (title + fifth read, correction);
|
||||
`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots);
|
||||
`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`;
|
||||
`systems/scheduled-tasks.md`; `current-state.md`; `index.md`.
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
# Wiki freshness audit — 2026-11-21
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (5)
|
||||
|
||||
- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30)
|
||||
- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30)
|
||||
- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30)
|
||||
|
||||
## FRESH (132)
|
||||
|
||||
- [ 10d] CONTRIBUTING.md
|
||||
- [ 13d] agents/hermes-chaos.md
|
||||
- [ 13d] agents/hermes-production.md
|
||||
- [ 10d] agents/index.md
|
||||
- [ 1d] agents/nanobot.md
|
||||
- [ 10d] agents/open-claw.md
|
||||
- [ 17d] agents/qwenpaw.md
|
||||
- [ 10d] agents/zeroclaw.md
|
||||
- [ 1d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 10d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 10d] comparisons/memory-backends.md
|
||||
- [ 10d] comparisons/model-routers.md
|
||||
- [ 1d] comparisons/steel-vs-camoufox.md
|
||||
- [ 10d] concepts/agent-memory.md
|
||||
- [ 1d] concepts/browser-automation.md
|
||||
- [ 10d] concepts/model-routing.md
|
||||
- [ 10d] concepts/persistent-shells.md
|
||||
- [ 10d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 1d] decisions/index.md
|
||||
- [ 10d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 10d] decisions/persistent-shell-enabled.md
|
||||
- [ 1d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 10d] experiments/index.md
|
||||
- [ 9d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 14d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 16d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 17d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 10d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 9d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 3d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 3d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 20d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 10d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 20d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 19d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 18d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 17d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 15d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 15d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 16d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 12d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 16d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 14d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 12d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 4d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 10d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 10d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 9d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 8d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 4d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||
- [ 0d] incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
|
||||
- [ 0d] incidents/2026-11-18-wiki-audit-credit-outage.md
|
||||
- [ 0d] incidents/2026-11-20-content-policy-blocked-digest-failures.md
|
||||
- [ 0d] index.md
|
||||
- [ 14d] infrastructure/backups.md
|
||||
- [ 10d] infrastructure/credentials-map.md
|
||||
- [ 14d] infrastructure/docker-services.md
|
||||
- [ 10d] infrastructure/domains-and-tunnels.md
|
||||
- [ 14d] infrastructure/hosts.md
|
||||
- [ 14d] infrastructure/network-map.md
|
||||
- [ 14d] infrastructure/proxmox.md
|
||||
- [ 9d] infrastructure/tailscale.md
|
||||
- [ 17d] queries/retained-research-answers.md
|
||||
- [ 2d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 20d] reports/uber-eats/2026-07.md
|
||||
- [ 20d] reports/uber-eats/2026-08.md
|
||||
- [ 20d] reports/uber-eats/2026-09.md
|
||||
- [ 8d] reports/uber-eats/2026-10.md
|
||||
- [ 8d] reports/uber-eats/index.md
|
||||
- [ 10d] runbooks/backup-wiki.md
|
||||
- [ 10d] runbooks/diagnose-docker.md
|
||||
- [ 10d] runbooks/gateway-resource-resilience.md
|
||||
- [ 14d] runbooks/job-radar.md
|
||||
- [ 10d] runbooks/multiplexer-setup.md
|
||||
- [ 3d] runbooks/provider-health.md
|
||||
- [ 1d] runbooks/recover-docker-service.md
|
||||
- [ 10d] runbooks/restart-browser.md
|
||||
- [ 10d] runbooks/restart-hermes.md
|
||||
- [ 10d] runbooks/restore-openrouter.md
|
||||
- [ 10d] runbooks/test-openrouter.md
|
||||
- [ 10d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 3d] runbooks/update-hermes-safely.md
|
||||
- [ 13d] systems/browser-backend.md
|
||||
- [ 10d] systems/cloudflare-tunnel.md
|
||||
- [ 4d] systems/conduit.md
|
||||
- [ 4d] systems/current-profile.md
|
||||
- [ 10d] systems/guanaco.md
|
||||
- [ 10d] systems/headless-server.md
|
||||
- [ 12d] systems/hermes-agent.md
|
||||
- [ 8d] systems/hermes-hq-push.md
|
||||
- [ 10d] systems/hermes-ui.md
|
||||
- [ 17d] systems/home-assistant.md
|
||||
- [ 17d] systems/ignis.md
|
||||
- [ 4d] systems/memory-backend.md
|
||||
- [ 19d] systems/messaging-integrations.md
|
||||
- [ 1d] systems/mnemosyne.md
|
||||
- [ 4d] systems/model-providers.md
|
||||
- [ 17d] systems/nanobot.md
|
||||
- [ 10d] systems/nextcloud.md
|
||||
- [ 10d] systems/obsidian.md
|
||||
- [ 11d] systems/omniroute.md
|
||||
- [ 10d] systems/opencode-go.md
|
||||
- [ 10d] systems/openrouter.md
|
||||
- [ 17d] systems/paseo.md
|
||||
- [ 6d] systems/phonon-tts.md
|
||||
- [ 11d] systems/push-notifications.md
|
||||
- [ 1d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 10d] systems/terminal-backend.md
|
||||
- [ 10d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages
|
||||
@@ -202,3 +202,30 @@ Primary agent memory store for Hermes.
|
||||
rather than fail. The **underlying cold start was not optimised** — only the budget was raised.
|
||||
- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic
|
||||
1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.
|
||||
|
||||
## Re-read 2026-11-21 (audit window 2026-11-20; live probes 2026-11-21 00:0x–00:5x AWST)
|
||||
|
||||
**The capture fault is *session-scoped*, and its boundary is exact — the 11-20 "conversational silence"
|
||||
reading is withdrawn.**
|
||||
|
||||
- **One session has stopped capturing entirely.** The desktop DM session `20261031_190813_95fbde` owns
|
||||
**124 Pulse/automation rows** in `working_memory` (84 under the bare id, 40 under
|
||||
`hermes_agent:main:local:dm:`), spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (= 13:10:35 AWST)**
|
||||
and **nothing after** — while the session itself kept producing user-role turns. Verified against
|
||||
`state.db`: last captured turn **13:06:05 AWST**, first uncaptured **13:21:00 AWST**, **~57 turns** since.
|
||||
The **11-13 13:14 gateway restart** sits inside that 15-minute boundary and is the leading hypothesis (a
|
||||
watermark left at-or-past the tip). **NOT PROVEN** — the oldest surviving `agent.log` begins
|
||||
**2026-11-13 19:03:34**, so the window has rotated out of the log horizon.
|
||||
- **Capture still works for *new* sessions.** The single write after that boundary —
|
||||
`2026-11-18 11:25:03 UTC = 19:25:03 AWST` — came from `hermes_20261118_191808_5c9e08`, a session created by a
|
||||
genuine prompt that evening. The pipeline is not globally dead; it fails for the long-lived session.
|
||||
- **Global silence now 52 h 35 m** (11-18 19:25:03 AWST → this read); exactly **one** write in eight days.
|
||||
`working_memory` **1,131** / `episodic_memory` **1,250** — both unchanged since the 11-20 read.
|
||||
- **The health monitor was already saying this.** `ea79a2107e36` fails with
|
||||
`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`; incident
|
||||
`ea79a2_ace0f263e18a` first seen 2026-11-21 00:03:12, alerted 00:03:17, unclosed.
|
||||
- **Consolidation still 0:** 199 of the last 200 cycles consolidated nothing.
|
||||
- **Conflicts still 59** against a threshold of 50; the watchdog has now been red for **210 consecutive
|
||||
15-minute slots** (11-18 20:01:15 → 11-21 00:31:16 (00:16:15 at the first read)).
|
||||
|
||||
→ [[incidents/2026-11-14-mnemosyne-capture-stall]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||
|
||||
@@ -1360,3 +1360,55 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
|
||||
condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both
|
||||
- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level.
|
||||
|
||||
## Re-verified 2026-11-21 (window 2026-11-20, AWST; live reads 2026-11-21 00:0x–00:5x)
|
||||
|
||||
**Fleet flat at 29 jobs / 26 enabled** (twelfth-plus consecutive window). Ledger, deliveries and the two
|
||||
standing incidents re-read; **two new failure signatures appeared on 11-20**, both raised by the model-facing
|
||||
API rather than by job logic.
|
||||
|
||||
### Non-clean `last_status`
|
||||
|
||||
| Job | ID | Schedule | Streak | Cause |
|
||||
|---|---|---|---|---|
|
||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | **210** | 59 unresolved conflicts vs threshold 50 (standing since 11-18 20:01) |
|
||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | **4** | `STALE MEMORY PIPELINE: capture missed: user message` |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | **1** | `content_policy_blocked` (11-20 08:31:06) |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | **1** | `content_policy_blocked` (11-20 12:01:14) |
|
||||
|
||||
All four are `enabled: true`. `failure_streak = 1` on the two digests means **isolated failures, not chronic** —
|
||||
the preceding slot of each completed (FreshRSS Midday ran clean on 11-19 12:01).
|
||||
|
||||
### New failure class: provider content filter
|
||||
|
||||
Both digests died on the identical string — `RuntimeError: content_policy_blocked: Output data may contain
|
||||
inappropriate content.` — hours apart, with every other job in the window clean. Both **alerted** (incident
|
||||
rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`, both unclosed) and both are classified
|
||||
**`failure_type: unknown`**, so the classifier cannot group or auto-close this class.
|
||||
→ [[incidents/2026-11-20-content-policy-blocked-digest-failures]]
|
||||
|
||||
### Standing reds
|
||||
|
||||
- **Watchdog: 210 consecutive 15-minute slots**, one stable incident id `96eb95_98ade422d195`
|
||||
(`first_seen_at` 11-18 20:01:15, `last_seen_at` 11-21 00:31:16 (00:16:15 at the first read), `alerted_at` 11-20 21:31:24, unclosed) —
|
||||
**no signature fragmentation**, contrary to the 11-20 page's concern.
|
||||
- **Health monitor: the opposite.** One standing condition re-forms a **fresh incident id every slot**
|
||||
(`ea79a2_f2be2db3bfce` 11-19 12:01:58, `ea79a2_aae4d914f3c9` 11-20 00:03:36, `ea79a2_ace0f263e18a`
|
||||
11-21 00:03:12) — three rows for one fault. The two monitors differ in de-duplication behaviour and it is
|
||||
worth knowing which is which when reading the ledger.
|
||||
- **Capture:** `working_memory` **1,131** / `episodic_memory` **1,250**, unchanged; corrected diagnosis on
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||
- **Consolidation 0 items** in 199 of the last 200 cycles.
|
||||
|
||||
### Freshness / files touched
|
||||
|
||||
- `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` →
|
||||
**5 stale / 131 fresh / 3 no-date of 139 pages**. The five at exactly 21 d are the predicted cohort:
|
||||
`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`.
|
||||
- **Pages touched by this audit:** [[incidents/2026-11-20-content-policy-blocked-digest-failures]] (new),
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] (title + fifth read),
|
||||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (115 → 210 slots),
|
||||
[[incidents/2026-11-18-wiki-audit-credit-outage]] (→ resolved), [[systems/mnemosyne]],
|
||||
[[current-state]], [[index]], `log.md`.
|
||||
- **No job added, retired, disabled or repinned; no page removed or archived; no system obsoleted; no decision
|
||||
reversed.**
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
updated: 2026-11-21
|
||||
verified_on: 2026-11-21
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user