From ca0ff5aa80310c43d72cb90ddaa4300bbf0a30dd Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sat, 21 Nov 2026 00:40:00 +0800 Subject: [PATCH] =?UTF-8?q?wiki=20audit=202026-11-21:=20session-scoped=20c?= =?UTF-8?q?apture=20re-diagnosis=20(124=20Pulse=20rows,=20~57=20uncaptured?= =?UTF-8?q?=20turns,=20gateway-restart=20hypothesis=20NOT=20proven);=20NEW?= =?UTF-8?q?=20page=20=E2=80=94=20first=20content=5Fpolicy=5Fblocked=20dige?= =?UTF-8?q?st=20failures=20(gmail=2008eb86f02d37=20+=20freshrss=20ecc1fc13?= =?UTF-8?q?655a);=20watchdog=20streak=20115->210=20(single=20incident=20ro?= =?UTF-8?q?w);=20wiki-audit=20credit=20outage=20RESOLVED=20(two=20nights?= =?UTF-8?q?=20lost,=2011-20=20delivery=20suppressed);=20ledger=20correctio?= =?UTF-8?q?ns=20(4=20unclosed=20health-monitor=20rows,=20not=205;=20audit?= =?UTF-8?q?=20incident=20closed=202026-11-20T01:19:13;=207=20unclosed=20cr?= =?UTF-8?q?on=5Fincidents);=20skills-index=20regenerated=20299=20skills/67?= =?UTF-8?q?=20cats;=20freshness=205=20stale/132=20fresh/3=20no-date;=20Qua?= =?UTF-8?q?rtz=20rebuilt+deployed=20(879=20files)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- current-state.md | 21 ++- .../2026-11-14-mnemosyne-capture-stall.md | 51 +++++- ...osyne-watchdog-conflict-threshold-storm.md | 28 +++- .../2026-11-18-wiki-audit-credit-outage.md | 33 +++- ...-content-policy-blocked-digest-failures.md | 64 ++++++++ index.md | 24 +-- log.md | 58 +++++++ raw/stocktake/freshness-20261121.txt | 153 ++++++++++++++++++ systems/mnemosyne.md | 27 ++++ systems/scheduled-tasks.md | 52 ++++++ systems/skills-index.md | 4 +- 11 files changed, 484 insertions(+), 31 deletions(-) create mode 100644 incidents/2026-11-20-content-policy-blocked-digest-failures.md create mode 100644 raw/stocktake/freshness-20261121.txt diff --git a/current-state.md b/current-state.md index 328528a..8bec483 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,27 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-11-20 -verified_on: 2026-11-20 -review_after: 2026-11-21 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-11-22 confidence: high tags: [system] -sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] +sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db, live-sqlite-mnemosyne] --- # Current State -Updated: 2026-11-20 +Updated: 2026-11-21 + +## Recent Changes (2026-11-20, nightly audit; live probes 2026-11-21 00:0x – 00:5x AWST) + +**Three things moved: the capture fault was re-diagnosed as session-scoped (correcting the 11-20 read), two digests died on a provider content filter, and the audit itself came back.** + +- **⚠ Capture — the 11-20 "every interactive session was captured" finding is WITHDRAWN.** That read counted *sessions*; capture is per *message*. Per message, the desktop DM session `20261031_190813_95fbde` owns **124** Pulse/automation rows in `working_memory`, spanning **2026-10-31 11:09:16 → 2026-11-13 13:10:35 AWST**, and **nothing after** — while the session kept producing turns: **~57** since (last captured turn **13:06:05**, first uncaptured **13:21:00**, verified in `state.db`). The **11-13 13:14 gateway restart** sits inside that 15-minute boundary — **leading hypothesis, explicitly NOT proven** (`agent.log`'s oldest surviving file begins **11-13 19:03:34**; the window has rotated out). Capture is **not** globally dead: the only write since, **`2026-11-18 19:25:03` AWST**, came from the new session `hermes_20261118_191808_5c9e08`. Counters frozen at `working_memory` **1,131** / `episodic_memory` **1,250**; silence since the last write **52 h 35 m**. The health monitor already reports it in its own words: `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak: 4`. → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]] +- **New failure class — `content_policy_blocked`.** Two unrelated digests died on the identical provider string on 11-20: **Gmail Action Digest** `08eb86f02d37` at **08:31:06** and **FreshRSS Midday News Briefing** `ecc1fc13655a` at **12:01:14**. Both `failure_streak: 1` (**isolated**), both **alerted**, both **unclosed**, both classified `failure_type: unknown`. → [[incidents/2026-11-20-content-policy-blocked-digest-failures]] +- **The audit came back — and the outage was two nights, not one.** `cbfe434a3b5e` reads `last_status: ok` / streak 0. Ledger: **11-18 00:00 failed**, **11-19 00:00 failed**, **11-20 00:00 completed** (report not delivered — `delivery_outcome: suppressed`), **11-21 00:00** this run. The 11-20 run absorbed windows **11-17 and 11-18** via its three-window catch-up. **No re-run was forced.** → [[incidents/2026-11-18-wiki-audit-credit-outage]] now `resolved`. +- **Watchdog still red — 210 consecutive 15-minute slots** (was 115 at the 11-20 read), **59 unresolved conflicts vs a threshold of 50**. Its incident `96eb95_98ade422d195` held **one id** from **11-18 20:01:15** to **11-21 00:31:16 (00:16:15 at the first read)** (**no fragmentation**) — unlike the **health monitor**, which re-forms a fresh incident id every slot for one standing condition (`ea79a2_f2be2db3bfce` → `…aae4d914f3c9` → `…ace0f263e18a`). → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] +- **Consolidation still zero-item:** 199 of the last 200 cycles. Fleet **29 jobs / 26 enabled** (flat). Freshness audit **5 stale / 131 fresh / 3 no-date of 139** — the five at exactly 21 d are the predicted cohort (`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`). +- **Pages touched:** [[incidents/2026-11-20-content-policy-blocked-digest-failures]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (title + fifth read), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (115 → 210), [[incidents/2026-11-18-wiki-audit-credit-outage]] (→ resolved), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-21**. ## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) diff --git a/incidents/2026-11-14-mnemosyne-capture-stall.md b/incidents/2026-11-14-mnemosyne-capture-stall.md index 524d795..431bb9d 100644 --- a/incidents/2026-11-14-mnemosyne-capture-stall.md +++ b/incidents/2026-11-14-mnemosyne-capture-stall.md @@ -1,13 +1,13 @@ --- -title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts +title: Mnemosyne capture is dead for the long-running desktop DM session since 2026-11-13 13:10 AWST — ~57 uncaptured turns, four unclosed health-monitor alerts type: incident status: active created: 2026-11-15 -updated: 2026-11-20 -verified_on: 2026-11-20 -review_after: 2026-11-21 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-11-22 confidence: high -tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate] +tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, session-scoped, freshness-gate] sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script] --- @@ -200,6 +200,47 @@ silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98a **Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days. +### Fifth read — 2026-11-21 00:0x–00:5x AWST: **the 11-20 conclusion above is withdrawn.** Capture is not "quiet", it is **dead for one specific session**, and the boundary is exact. + +The 11-20 read counted **sessions** in `state.db`, found only three in the window, and concluded that every +interactive session had been captured — so the alert was measuring conversational silence. **The wrong unit.** +Capture is per *message*. Counted per message, the picture inverts: + +| | | +|---|---| +| Rows this session owns in `working_memory` | **124** (84 under the bare id `hermes_20261031_190813_95fbde`, 40 under `hermes_agent:main:local:dm:20261031_190813_95fbde`) | +| Their span | **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC = 2026-11-13 13:10:35 AWST** — then nothing | +| Last *captured* turn in `state.db` | **2026-11-13 13:06:05 AWST** (Pulse report `pc36e5568`, row written 13:10:35) | +| First *uncaptured* turn | **2026-11-13 13:21:00 AWST** (`Pulse delivery check p65be4717`) — and every turn since | +| Uncaptured turns since | **~57** (11-13 19:21 → 11-21 00:00): Pulse reports, delivery checks, morning briefs, `[System: …]` notices | +| Writes from this session after that boundary | **0** | + +- **The break sits in a 15-minute gap, and the 11-13 13:14 gateway restart sits inside it.** Last captured turn + 13:06:05, first uncaptured 13:21:00; the 11-13/11-14 audit record documents a **gateway restart at 13:14** + that day (the one that cleared the stale `TELEGRAM_HOME_CHANNEL_THREAD_ID`). *Leading hypothesis:* the + restart left this session's capture watermark at-or-past its tip, so the session syncs nothing while new + sessions start clean. **NOT PROVEN** — the oldest surviving `agent.log` file begins **2026-11-13 19:03:34**, + so the 13:14 window has rotated out of the log horizon and no startup banner is recoverable. +- **Capture is not globally dead — it works for *new* sessions.** The single write after that boundary, + `2026-11-18 11:25:03 UTC = 19:25:03 AWST`, came from `hermes_20261118_191808_5c9e08`, a session created by a + genuine prompt that evening. That is the **only** write in eight days. +- **The health monitor already says exactly this,** in its own words: `ea79a2107e36` is failing with + `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`. The gate was right; the 11-20 + reading of it was wrong. + +**Corrected statement for the record:** + +> **Mnemosyne has captured nothing from the desktop DM session `20261031_190813_95fbde` since 2026-11-13 +> 13:10:35 AWST — ~8 days and ~57 user-role turns — while capturing normally from sessions created after that +> date.** + +**Counters (11-21 read):** `working_memory` **1,131** / `episodic_memory` **1,250** — both **unchanged** from +the 11-20 read; global silence since the last write **52 h 35 m**. **Unclosed incidents: 4 → 5** +(`cbfe43_fa12cca1ad4b` auth, `alerted_at: None`; `96eb95_98ade422d195` script, still alerting 11-21 00:31:16 (00:16:15 at the first read); +`ea79a2_f2be2db3bfce` 11-19 12:01:58; `ea79a2_aae4d914f3c9` 11-20 00:03:36; **new** `ea79a2_ace0f263e18a` +11-21 00:03:12). **Next probe:** replay one Pulse-shaped turn into that session with the provider's debug +logging on, and diff against a pre-11-13 capture in `agent.log`. + ## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST) - Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e. diff --git a/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md b/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md index c2868a1..e4da997 100644 --- a/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md +++ b/incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md @@ -1,22 +1,22 @@ --- -title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST) +title: Mnemosyne Watchdog red for 210 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST) type: incident status: active created: 2026-11-20 -updated: 2026-11-20 -verified_on: 2026-11-20 -review_after: 2026-11-21 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-11-22 confidence: high tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident] sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json] --- -# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST) +# Mnemosyne Watchdog red for 210 consecutive slots (from 2026-11-18 20:01 AWST) — 115 at the 2026-11-20 read **Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since **2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit). -`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**. +`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`** *(that is the **2026-11-20** reading — this page was written then; the re-read below carries the streak to **210**, `last_seen_at` **2026-11-21 00:31:16**)*. This is the estate's **only** red cron row with a large streak, and it is a *single durable condition* re-reported every 15 minutes, not a series of separate faults. @@ -99,6 +99,22 @@ LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** th noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis — the fact pairs were not inspected. +## Re-read 2026-11-21 00:0x–00:5x AWST — 210 consecutive slots; **no signature fragmentation** + +- `failure_streak` = **210** (was 115 at the 11-20 read; the job's cadence is 96 slots/day, so this is + 11-18 20:01 → 11-21 00:16 ≈ 2 days 4 h of continuous red). +- **Incident `96eb95_98ade422d195` is one row, not a series:** `first_seen_at` **2026-11-18 20:01:15**, + `last_seen_at` **2026-11-21 00:31:16 (00:16:15 at the first read)**, `alerted_at` **2026-11-20 21:31:24**, `closed_at` empty. The 11-20 + page raised signature fragmentation as a risk; on this evidence the watchdog's de-duplication **holds**. +- **The contrast is the health monitor**, which re-forms a **new incident id every slot** for one standing + condition (`ea79a2_f2be2db3bfce` 11-19 12:01:58 → `ea79a2_aae4d914f3c9` 11-20 00:03:36 → + `ea79a2_ace0f263e18a` 11-21 00:03:12). Same estate, same period, opposite behaviour — worth knowing when + reading the ledger for "new" versus "chronic". +- Error text unchanged: **59 unresolved conflicts against a threshold of 50**. Nothing was resolved, + reclassified or deleted at this read either. +- **Still the standing question:** a *level*-based test re-reports on every slot. A *state-change* alert would + have fired once on 11-18 20:01 and then stayed quiet. + ## Not applied / open - **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed. diff --git a/incidents/2026-11-18-wiki-audit-credit-outage.md b/incidents/2026-11-18-wiki-audit-credit-outage.md index 5ebab87..90a51b2 100644 --- a/incidents/2026-11-18-wiki-audit-credit-outage.md +++ b/incidents/2026-11-18-wiki-audit-credit-outage.md @@ -1,11 +1,11 @@ --- title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited type: incident -status: active +status: resolved created: 2026-11-20 -updated: 2026-11-20 -verified_on: 2026-11-20 -review_after: 2026-11-21 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-11-28 confidence: high tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure] sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log] @@ -74,6 +74,31 @@ skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path rather than at the alert throttle. +## Re-read 2026-11-21 00:0x AWST — **resolved**; and the ledger shows the outage was two nights, not one + +- **The job is clean again.** `cbfe434a3b5e` reads `last_status: ok`, `failure_streak = 0`. The execution ledger + dates the outage exactly: + + | Slot | Execution | Outcome | + |---|---|---| + | 2026-11-18 00:00:08 | `d6323ba3b7ca4d8a95d0f57bf20b4807` | **failed** (`delivery_outcome: suppressed`) | + | 2026-11-19 00:00:16 | `668984226ccc47b09cb66fe65ae89ddc` | **failed** (`delivery_outcome: suppressed`) | + | 2026-11-20 00:00:08 | `b88bd7d8007248399f57cccfc98b536a` | **completed** (report not delivered) | + | 2026-11-21 00:00:08 | `ec65ca362c6847fcb9405daa15d1f6ed` | running — the run authoring this note | + +- **Two nights were lost, not one.** The 11-18 and 11-19 runs both failed, so windows **11-17 and 11-18** have + no audit of their own. The **11-20 run absorbed both** via its three-window catch-up (2026-11-17 → 2026-11-20), + read from the cron ledger and `state.db` rather than from conversation — `state.db` holds **no** interactive + sessions in those windows, so there was no conversation to miss. +- **No re-run of either lost slot was forced**; recovery came from the next scheduled slot running normally. +- **Not re-tested:** whether the credit/auth fault can recur. The job's pin, provider and schedule are unchanged + and no credential was rotated, so a recurrence would present identically. +- **The silent-skip class this page identified is now observed, not inferred:** both failed nights carry + `delivery_outcome: suppressed` and the incident row `cbfe43_fa12cca1ad4b` has `alerted_at: NULL` — two + consecutive failed nights produced **no alert and no report**, which is why the outage only surfaced when the + 11-20 audit read the ledger. Whether that suppression is job-level config or framework behaviour is still + **UNVERIFIED** (the job's delivery config was not read). + ## Not applied / open - **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no diff --git a/incidents/2026-11-20-content-policy-blocked-digest-failures.md b/incidents/2026-11-20-content-policy-blocked-digest-failures.md new file mode 100644 index 0000000..4da0f1f --- /dev/null +++ b/incidents/2026-11-20-content-policy-blocked-digest-failures.md @@ -0,0 +1,64 @@ +--- +title: Two digests died on `content_policy_blocked` (2026-11-20) — first content-filter failures on record +type: incident +status: active +created: 2026-11-21 +updated: 2026-11-21 +verified_on: 2026-11-21 +review_after: 2026-11-22 +confidence: high +tags: [cron, content-policy, provider-error, gmail, freshrss, digests, incident, unclassified-failure] +sources: [live-probe-cron-executions-db-2026-11-21, live-probe-cron-incidents-2026-11-21, live-probe-jobs-json-2026-11-21] +--- + +# Two digests died on `content_policy_blocked` (2026-11-20) + +**The first content-filter failures in the retained log.** Two unrelated jobs, on two different feeds, died +with the *identical* terminal string on 2026-11-20 — hours apart, with every other job in the window clean. + +## What happened + +| Job | Job ID | Slot (AWST) | Execution | Duration | Terminal error | +|---|---|---|---|---|---| +| Gmail Action Digest | `08eb86f02d37` | 08:31:06 → 08:32:45 | `a490b74a509e4303acad671ee1ee80cf` | 1m39s | `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` | +| FreshRSS Midday News Briefing | `ecc1fc13655a` | 12:01:14 → 12:05:04 | `0e1c11f49313428799de604afe73595c` | 3m50s | same string | + +Both jobs are pinned `provider: custom:omni-101` / `model: smart-route`. Both carry +**`failure_streak = 1`** — i.e. **isolated single failures, not chronic**: the preceding slot of each job +completed. FreshRSS Midday ran clean on 11-19 12:01. + +## What is established + +- The failure is raised **by the model-facing API, not by the job logic**. The string is a provider content + filter surfaced to the job as a wrapped `RuntimeError`; the digest scripts assembled their input fine. +- **Both jobs alerted, and both alerts are unclosed.** Incident rows: + - `08eb86_c0cca1a55b96` — first seen 11-20 08:32:44, `alerted_at` 08:32:46 + - `ecc1fc_5a10b70830d1` — first seen 11-20 12:05:03, `alerted_at` 12:05:05 + + This is the **opposite** of [[incidents/2026-11-18-wiki-audit-credit-outage]], which failed silently for two + nights: here the alerting path worked end to end. +- `failure_type` is recorded as **`unknown`** for both. The incident classifier does not recognise this + class, so it can neither group nor auto-close it — a recurrence will alert, but will not be understood. +- Both left a full transcript: `~/.hermes/cron/output/08eb86f02d37/2026-11-20_08-32-44.md` and + `~/.hermes/cron/output/ecc1fc13655a/2026-11-20_12-05-03.md`. + +## What is NOT established + +- **Whether the trigger was the request or the response.** The message reads "Output data may contain + inappropriate content", but a filter that scans a request payload reports the same way. The Gmail digest + reads mail; the FreshRSS digest reads news — the shared factor is more likely a *token* the router's filter + dislikes than the subject matter. **The two output files above settle this**: diff what the two prompts had + in common. +- **Whether the router (`omni-101`) applies a filter its upstreams do not.** If it does, the same feeds can + fail again with nothing changed on this host. Probe: replay one of the two prompts against the router and + against an upstream provider directly, and compare. +- Whether it recurs. Two slots, two jobs, one day is a pattern worth watching — not yet a trend. + +## Standing risk + +Two of the estate's daily digests sit behind a provider filter that has already rejected them once, and the +classifier files the failure as `unknown` — so a recurrence will alert without being recognised. Cheapest +mitigation if it recurs: **no content-policy string is worth a lost digest** — route the two digest jobs to a +provider without the filter, or strip the highest-risk section from the prompt payload. + +→ [[systems/scheduled-tasks]], [[systems/mnemosyne]] diff --git a/index.md b/index.md index 04f33ae..998a192 100644 --- a/index.md +++ b/index.md @@ -3,16 +3,21 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-20 -verified_on: 2026-11-20 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high -tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] -sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] +tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout, content-policy, session-scoped-capture] +sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, live-sqlite-mnemosyne, raw/stocktake] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. last updated: 2026-11-20 +> model routing and automation. last updated: 2026-11-21 +> +> **2026-11-21 (nightly audit; window 2026-11-20; live probes 2026-11-21 00:0x – 00:5x AWST) — the memory fault is session-scoped, a new provider content filter killed two digests, and the audit came back.** +> **⚠ Capture, corrected a second time.** The 11-20 finding that "**every** interactive session was captured" is **withdrawn** — that read counted *sessions*; capture is per *message*. Per message: the desktop DM session `20261031_190813_95fbde` owns **124** Pulse/automation rows in `working_memory` from **2026-10-31 11:09:16** to **2026-11-13 13:10:35 AWST** and **nothing after**, while the session kept producing turns — **~57** since, last captured turn **13:06:05**, first uncaptured **13:21:00** (verified in `state.db`). The **11-13 13:14 gateway restart** sits inside that 15-minute boundary — **leading hypothesis, explicitly NOT proven** (`agent.log`'s oldest surviving file begins **11-13 19:03:34**; the window has rotated out). Capture is **not** globally dead: the only write since, **`2026-11-18 19:25:03` AWST**, came from the new session `hermes_20261118_191808_5c9e08`. Counters frozen at `working_memory` **1,131** / `episodic_memory` **1,250**; **52 h 35 m** since the last write; the health monitor already says it — `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak: 4` → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]. +> **New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string on 11-20 — **Gmail Action Digest `08eb86f02d37` 08:31:06** and **FreshRSS Midday News Briefing `ecc1fc13655a` 12:01:14** — `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both `failure_streak: 1` (**isolated**, not chronic), both **alerted** and **unclosed**, both `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]]. +> **The audit came back — and the outage was two nights, not one.** `cbfe434a3b5e` is `ok` / streak 0; the ledger dates it **11-18 00:00 failed · 11-19 00:00 failed · 11-20 00:00 completed** (report not delivered — `delivery_outcome: suppressed`) **· 11-21 00:00** this run. The 11-20 run absorbed windows **11-17/11-18** via its three-window catch-up; **no re-run was forced** → [[incidents/2026-11-18-wiki-audit-credit-outage]] now **`resolved`**. The **Mnemosyne Watchdog is still red — 210 consecutive 15-minute slots** (59 conflicts vs threshold 50), holding **one** incident id `96eb95_98ade422d195` from **11-18 20:01:15** to **11-21 00:31:16 (00:16:15 at the first read)** (**no fragmentation**), whereas the **health monitor** re-forms a fresh incident id every slot for one standing condition → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. Consolidation **0 items** (199 of the last 200 cycles). Fleet **29 jobs / 26 enabled**. Freshness **5 stale / 131 fresh / 3 no-date** of 139. > > **2026-11-20 (three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) — the memory freeze broke by itself, the alarm did not, and the audits that would have said so had died twice.** > **The capture stall is over — silently.** `working_memory` **1130 → 1131** / `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST**: **105 h 52 m**, ended with **no intervention**. And the "capture is dead" reading was wrong — `state.db` holds only **three** non-cron sessions in that window (two `desktop` on 11-14, both *before* the 09:33 boundary, then the 11-18 **19:18** session whose turn is the captured row), so **every interactive session was captured**; the alert measures **conversational silence** (age of the newest non-`cron`/`subagent` message). Corrected on [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]. @@ -93,11 +98,12 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **OPEN — the nightly audit failed on two consecutive nights, so 2026-11-17 and 2026-11-18 were never audited, and no failure notice was ever sent.** `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`** — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. The 2026-11-20 00:00 slot (this run) covers all three windows. -- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 115 consecutive watchdog failures from one cause: the conflict queue.** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 115`** at **2026-11-20 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 115 faults. Each slot delivers a *failure notice* (9 of the 24 deliveries since 11-18 are `for_failure: 1`), so the noisy channel is also the only channel; every slot re-opens incident `96eb95_98ade422d195` and nothing escalates or closes. -- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (corrected 2026-11-20) — the freeze ended by itself at 2026-11-18 19:25 AWST after 105 h 52 m, and the "captured nothing" reading was a quiet estate, not a broken hook.** Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. +- [[incidents/2026-11-20-content-policy-blocked-digest-failures]] - **OPEN — first `content_policy_blocked` failures on record: two unrelated digests killed by the identical provider string hours apart.** **Gmail Action Digest** `08eb86f02d37` ran **2026-11-20 08:31:06 → 08:32:45** and **FreshRSS Midday News Briefing** `ecc1fc13655a` ran from **12:01:14**, both ending `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both carry **`failure_streak: 1`** (isolated — each job's preceding slot completed), both **alerted** (incident rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`) and both still **unclosed**, and both are classified **`failure_type: unknown`**, so the classifier cannot group the class or auto-close it. Nothing was changed on either job and **no re-run was forced**; whether the class recurs is unknown. +- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **RESOLVED 2026-11-21 — the nightly audit failed on two consecutive nights, so windows 2026-11-17 and 2026-11-18 were never audited on their own night, and no failure notice was ever sent.** The ledger dates it exactly: `cbfe434a3b5e` failed **2026-11-18T00:00:08** and **2026-11-19T00:00:16** on a chained-provider **credit/auth** fault, then **completed on 11-20T00:00:08** (report not delivered — `delivery_outcome: suppressed`), absorbing both lost windows via its three-window catch-up; the job now reads `last_status: ok` / `failure_streak: 0`. **No re-run was forced**, and the silent-skip class it identified stays open on [[systems/scheduled-tasks]]. `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) was **never alerted** (`alerted_at: None`) — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] — and is now **resolved** (`closed_at` **2026-11-20 01:19:13**, i.e. it cleared itself on the first clean completion). +- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 210 consecutive watchdog failures from one cause: the conflict queue (115 at the 11-20 read).** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 210`** at **2026-11-21 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 210 faults. Each slot delivers a *failure notice*, so the noisy channel is also the only channel. Unlike the health monitor, this job **de-duplicates**: **one** incident row `96eb95_98ade422d195` (`failure_type: script`) has carried all 210 slots — `first_seen` 2026-11-18 20:01:15, `last_seen` **2026-11-21 00:31:16**, `alerted_at` 2026-11-20 21:31:24, still **unclosed** — so nothing escalates and nothing closes. +- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (re-corrected 2026-11-21) — the 11-20 "quiet estate, not a broken hook" reading is WITHDRAWN.** Counted per *message* rather than per *session*, the desktop DM session `20261031_190813_95fbde` owns **124** `working_memory` rows ending **2026-11-13 13:10:35 AWST** with **~57** of its turns uncaptured since (last captured turn 13:06:05, first uncaptured 13:21:00; the **11-13 13:14 gateway restart** sits inside that gap — leading hypothesis, not proven). The **105 h 52 m** global freeze did end by itself at **2026-11-18 19:25 AWST** and capture still works for **new** sessions; the failing one is the long-lived desktop session. Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. - [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). -- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). +- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion — its incident row was closed 2026-11-12T00:25:36; *cron_incidents* stood at 137 rows at that read) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and was **never alerted** (`alerted_at: NULL`) and is now **resolved** (`closed_at` **2026-11-12T00:25:36**) — so "the estate's only open incident row" has been overtaken: unclosed *cron_incidents* at the **2026-11-21** read is **7** (`08eb86` ×1, `ecc1fc` ×1, `ea79a2` ×4, `96eb95` ×1). Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]] - [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed. - [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible. diff --git a/log.md b/log.md index ccdd404..dd01f90 100644 --- a/log.md +++ b/log.md @@ -1890,3 +1890,61 @@ recorded here so the next audit does not re-litigate it. - `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27). - **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists). - Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced. + +--- + +## 2026-11-21 00:0x–00:5x AWST — nightly wiki audit (window: 2026-11-20) + +**Headline: the memory capture fault is *session-scoped*, and its boundary is exact — the 11-20 +"conversational silence" conclusion is withdrawn.** + +The desktop DM session `20261031_190813_95fbde` owns **124 Pulse/automation rows** in `working_memory`, +spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (13:10:35 AWST)** and **nothing after** — while the +session itself kept producing turns: **~57** since, last captured turn **13:06:05 AWST**, first uncaptured +**13:21:00 AWST** (verified against `state.db`). The **11-13 13:14 gateway restart** sits inside that +15-minute boundary; that is the **leading hypothesis, explicitly NOT proven** — the oldest surviving +`agent.log` file begins **11-13 19:03:34**, so the window has rotated out of the log horizon. Capture is +**not** globally dead: the single write since, `2026-11-18 19:25:03 AWST`, came from the new session +`hermes_20261118_191808_5c9e08`. Global silence now **52 h 35 m**; counters frozen at `working_memory` +**1,131** / `episodic_memory` **1,250**. The health monitor is already reporting this in its own words — +`STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`. + +**New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string +on 11-20 — Gmail Action Digest **08:31:06** (`08eb86f02d37`) and FreshRSS Midday News Briefing **12:01:14** +(`ecc1fc13655a`), both `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` +Both carry `failure_streak = 1` (**isolated**, not chronic), both **alerted** and both **unclosed**, both +classified `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]]. + +**Watchdog: 210 consecutive 15-minute slots** (was 115 at the 11-20 read). Incident +`96eb95_98ade422d195` held **one id** from 11-18 20:01:15 to 11-21 00:31:16 (00:16:15 at the first read) — **no fragmentation**, unlike the +health monitor, which re-forms a fresh incident id every slot for a single standing condition. 59 conflicts +against a threshold of 50. Consolidation still **0 items** (199 of the last 200 cycles). + +**The wiki-audit outage is resolved.** `cbfe434a3b5e` now reads `last_status: ok` / `failure_streak: 0`, and +the execution ledger dates the outage exactly: **11-18 00:00:08 failed**, **11-19 00:00:16 failed**, **11-20 +00:00:08 completed** (report not delivered — `delivery_outcome: suppressed`), and **11-21 00:00:08** is this +run. So **two** nights were lost (windows 11-17 and 11-18); the 11-20 run absorbed both via its three-window +catch-up, which read the cron ledger and `state.db` rather than conversation. **No re-run was forced.** +→ [[incidents/2026-11-18-wiki-audit-credit-outage]] now `resolved`. + +**Ledger confirmation read at the end of this run** (`~/.hermes/cron/executions.db`, `cron_incidents`, 2026-11-21 00:3x): +its incident row `cbfe43_fa12cca1ad4b` (`auth`) was **never alerted** and is **closed** (`closed_at` **2026-11-20 01:19:13**) — it +cleared itself on the first clean completion. Unclosed rows stand at **7**, and none of them belong to the audit: +`08eb86_c0cca1a55b96` + `ecc1fc_5a10b70830d1` (the two `content_policy_blocked` digests, `unknown`), +four `ea79a2_*` health-monitor rows (`auth`, one per slot: `f2be2db3bfce`, `aae4d914f3c9`, `51c605d08ea5`, `ace0f263e18a`), +and `96eb95_98ade422d195` (`script`, the conflict-queue trip). The same read moved the watchdog's `last_seen_at` +to **00:31:16** — the streak is still live, so every "hold one id" timestamp on these pages is a moving target; +the pages now cite **00:31:16 (00:16:15 at the first read)**. Earlier standing claims corrected in the same pass: +the health monitor's unclosed count is **four** (not five) — the capture-stall title now says so — and +[[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]'s "estate's only open incident row" is overtaken. + +**Freshness:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` → +**5 stale / 131 fresh / 3 no-date of 139 pages**; the five at exactly 21 d are the predicted cohort +(`job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`). + +**No job added, retired, disabled or repinned; no wiki page removed or archived; no decision reversed.** +Pages touched: new `incidents/2026-11-20-content-policy-blocked-digest-failures.md`; +`incidents/2026-11-14-mnemosyne-capture-stall.md` (title + fifth read, correction); +`incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md` (115 → 210 slots); +`incidents/2026-11-18-wiki-audit-credit-outage.md` (→ resolved); `systems/mnemosyne.md`; +`systems/scheduled-tasks.md`; `current-state.md`; `index.md`. diff --git a/raw/stocktake/freshness-20261121.txt b/raw/stocktake/freshness-20261121.txt new file mode 100644 index 0000000..35bfcfd --- /dev/null +++ b/raw/stocktake/freshness-20261121.txt @@ -0,0 +1,153 @@ +# Wiki freshness audit — 2026-11-21 +Threshold: >21 days since `updated` + +## STALE (5) + +- [ 22d] incidents/2026-10-27-job-radar-budget-exhausted.md (updated 2026-10-30) +- [ 22d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md (updated 2026-10-30) +- [ 22d] incidents/2026-10-29-conduit-voice-silence.md (updated 2026-10-30) +- [ 22d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md (updated 2026-10-30) +- [ 22d] systems/homepage-dashboard.md (updated 2026-10-30) + +## FRESH (132) + +- [ 10d] CONTRIBUTING.md +- [ 13d] agents/hermes-chaos.md +- [ 13d] agents/hermes-production.md +- [ 10d] agents/index.md +- [ 1d] agents/nanobot.md +- [ 10d] agents/open-claw.md +- [ 17d] agents/qwenpaw.md +- [ 10d] agents/zeroclaw.md +- [ 1d] comparisons/camoufox-vs-chrome.md +- [ 10d] comparisons/hermes-vs-nanobot.md +- [ 10d] comparisons/memory-backends.md +- [ 10d] comparisons/model-routers.md +- [ 1d] comparisons/steel-vs-camoufox.md +- [ 10d] concepts/agent-memory.md +- [ 1d] concepts/browser-automation.md +- [ 10d] concepts/model-routing.md +- [ 10d] concepts/persistent-shells.md +- [ 10d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 1d] decisions/index.md +- [ 10d] decisions/keep-openrouter-as-fallback.md +- [ 10d] decisions/persistent-shell-enabled.md +- [ 1d] decisions/rollback-steel-to-camofox.md +- [ 10d] experiments/index.md +- [ 9d] incidents/2026-07-29-mnemosyne-relay.md +- [ 10d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 9d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 14d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 16d] incidents/2026-08-15-skills-index-drift.md +- [ 17d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 10d] incidents/2026-08-29-gmail-read-state.md +- [ 9d] incidents/2026-09-02-sftpgo-outage.md +- [ 17d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 3d] incidents/2026-10-03-ct450-hermes-update.md +- [ 3d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 1d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 6d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 20d] incidents/2026-10-09-memory-recall-relevance.md +- [ 15d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 3d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 11d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 3d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 9d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 6d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 10d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 2d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 3d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 2d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 21d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 21d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 20d] incidents/2026-10-31-asana-access-failures.md +- [ 20d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 20d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 19d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 18d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 17d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 15d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 15d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 16d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 12d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 16d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 14d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 12d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 4d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +- [ 10d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md +- [ 10d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md +- [ 9d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md +- [ 8d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md +- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md +- [ 4d] incidents/2026-11-15-plugin-load-timeout-storm.md +- [ 0d] incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md +- [ 0d] incidents/2026-11-18-wiki-audit-credit-outage.md +- [ 0d] incidents/2026-11-20-content-policy-blocked-digest-failures.md +- [ 0d] index.md +- [ 14d] infrastructure/backups.md +- [ 10d] infrastructure/credentials-map.md +- [ 14d] infrastructure/docker-services.md +- [ 10d] infrastructure/domains-and-tunnels.md +- [ 14d] infrastructure/hosts.md +- [ 14d] infrastructure/network-map.md +- [ 14d] infrastructure/proxmox.md +- [ 9d] infrastructure/tailscale.md +- [ 17d] queries/retained-research-answers.md +- [ 2d] reports/communities-wa-social-media-audit-2026.md +- [ 20d] reports/uber-eats/2026-07.md +- [ 20d] reports/uber-eats/2026-08.md +- [ 20d] reports/uber-eats/2026-09.md +- [ 8d] reports/uber-eats/2026-10.md +- [ 8d] reports/uber-eats/index.md +- [ 10d] runbooks/backup-wiki.md +- [ 10d] runbooks/diagnose-docker.md +- [ 10d] runbooks/gateway-resource-resilience.md +- [ 14d] runbooks/job-radar.md +- [ 10d] runbooks/multiplexer-setup.md +- [ 3d] runbooks/provider-health.md +- [ 1d] runbooks/recover-docker-service.md +- [ 10d] runbooks/restart-browser.md +- [ 10d] runbooks/restart-hermes.md +- [ 10d] runbooks/restore-openrouter.md +- [ 10d] runbooks/test-openrouter.md +- [ 10d] runbooks/troubleshoot-tool-loop.md +- [ 3d] runbooks/update-hermes-safely.md +- [ 13d] systems/browser-backend.md +- [ 10d] systems/cloudflare-tunnel.md +- [ 4d] systems/conduit.md +- [ 4d] systems/current-profile.md +- [ 10d] systems/guanaco.md +- [ 10d] systems/headless-server.md +- [ 12d] systems/hermes-agent.md +- [ 8d] systems/hermes-hq-push.md +- [ 10d] systems/hermes-ui.md +- [ 17d] systems/home-assistant.md +- [ 17d] systems/ignis.md +- [ 4d] systems/memory-backend.md +- [ 19d] systems/messaging-integrations.md +- [ 1d] systems/mnemosyne.md +- [ 4d] systems/model-providers.md +- [ 17d] systems/nanobot.md +- [ 10d] systems/nextcloud.md +- [ 10d] systems/obsidian.md +- [ 11d] systems/omniroute.md +- [ 10d] systems/opencode-go.md +- [ 10d] systems/openrouter.md +- [ 17d] systems/paseo.md +- [ 6d] systems/phonon-tts.md +- [ 11d] systems/push-notifications.md +- [ 1d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 10d] systems/terminal-backend.md +- [ 10d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 5 stale, 132 fresh, 3 no-date of 140 pages \ No newline at end of file diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 30bb087..f878752 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -202,3 +202,30 @@ Primary agent memory store for Hermes. rather than fail. The **underlying cold start was not optimised** — only the budget was raised. - **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window. + +## Re-read 2026-11-21 (audit window 2026-11-20; live probes 2026-11-21 00:0x–00:5x AWST) + +**The capture fault is *session-scoped*, and its boundary is exact — the 11-20 "conversational silence" +reading is withdrawn.** + +- **One session has stopped capturing entirely.** The desktop DM session `20261031_190813_95fbde` owns + **124 Pulse/automation rows** in `working_memory` (84 under the bare id, 40 under + `hermes_agent:main:local:dm:`), spanning **2026-10-31 11:09:16 → 2026-11-13 05:10:35 UTC (= 13:10:35 AWST)** + and **nothing after** — while the session itself kept producing user-role turns. Verified against + `state.db`: last captured turn **13:06:05 AWST**, first uncaptured **13:21:00 AWST**, **~57 turns** since. + The **11-13 13:14 gateway restart** sits inside that 15-minute boundary and is the leading hypothesis (a + watermark left at-or-past the tip). **NOT PROVEN** — the oldest surviving `agent.log` begins + **2026-11-13 19:03:34**, so the window has rotated out of the log horizon. +- **Capture still works for *new* sessions.** The single write after that boundary — + `2026-11-18 11:25:03 UTC = 19:25:03 AWST` — came from `hermes_20261118_191808_5c9e08`, a session created by a + genuine prompt that evening. The pipeline is not globally dead; it fails for the long-lived session. +- **Global silence now 52 h 35 m** (11-18 19:25:03 AWST → this read); exactly **one** write in eight days. + `working_memory` **1,131** / `episodic_memory` **1,250** — both unchanged since the 11-20 read. +- **The health monitor was already saying this.** `ea79a2107e36` fails with + `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak = 4`; incident + `ea79a2_ace0f263e18a` first seen 2026-11-21 00:03:12, alerted 00:03:17, unclosed. +- **Consolidation still 0:** 199 of the last 200 cycles consolidated nothing. +- **Conflicts still 59** against a threshold of 50; the watchdog has now been red for **210 consecutive + 15-minute slots** (11-18 20:01:15 → 11-21 00:31:16 (00:16:15 at the first read)). + +→ [[incidents/2026-11-14-mnemosyne-capture-stall]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 5f17858..99e609c 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -1360,3 +1360,55 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes: condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both - **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level. + +## Re-verified 2026-11-21 (window 2026-11-20, AWST; live reads 2026-11-21 00:0x–00:5x) + +**Fleet flat at 29 jobs / 26 enabled** (twelfth-plus consecutive window). Ledger, deliveries and the two +standing incidents re-read; **two new failure signatures appeared on 11-20**, both raised by the model-facing +API rather than by job logic. + +### Non-clean `last_status` + +| Job | ID | Schedule | Streak | Cause | +|---|---|---|---|---| +| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | **210** | 59 unresolved conflicts vs threshold 50 (standing since 11-18 20:01) | +| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | **4** | `STALE MEMORY PIPELINE: capture missed: user message` | +| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | **1** | `content_policy_blocked` (11-20 08:31:06) | +| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | **1** | `content_policy_blocked` (11-20 12:01:14) | + +All four are `enabled: true`. `failure_streak = 1` on the two digests means **isolated failures, not chronic** — +the preceding slot of each completed (FreshRSS Midday ran clean on 11-19 12:01). + +### New failure class: provider content filter + +Both digests died on the identical string — `RuntimeError: content_policy_blocked: Output data may contain +inappropriate content.` — hours apart, with every other job in the window clean. Both **alerted** (incident +rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`, both unclosed) and both are classified +**`failure_type: unknown`**, so the classifier cannot group or auto-close this class. +→ [[incidents/2026-11-20-content-policy-blocked-digest-failures]] + +### Standing reds + +- **Watchdog: 210 consecutive 15-minute slots**, one stable incident id `96eb95_98ade422d195` + (`first_seen_at` 11-18 20:01:15, `last_seen_at` 11-21 00:31:16 (00:16:15 at the first read), `alerted_at` 11-20 21:31:24, unclosed) — + **no signature fragmentation**, contrary to the 11-20 page's concern. +- **Health monitor: the opposite.** One standing condition re-forms a **fresh incident id every slot** + (`ea79a2_f2be2db3bfce` 11-19 12:01:58, `ea79a2_aae4d914f3c9` 11-20 00:03:36, `ea79a2_ace0f263e18a` + 11-21 00:03:12) — three rows for one fault. The two monitors differ in de-duplication behaviour and it is + worth knowing which is which when reading the ledger. +- **Capture:** `working_memory` **1,131** / `episodic_memory` **1,250**, unchanged; corrected diagnosis on + [[incidents/2026-11-14-mnemosyne-capture-stall]]. +- **Consolidation 0 items** in 199 of the last 200 cycles. + +### Freshness / files touched + +- `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261121.txt 21` → + **5 stale / 131 fresh / 3 no-date of 139 pages**. The five at exactly 21 d are the predicted cohort: + `job-radar` ×2, `conduit-voice`, `m93p-nfs`, `homepage-dashboard`. +- **Pages touched by this audit:** [[incidents/2026-11-20-content-policy-blocked-digest-failures]] (new), + [[incidents/2026-11-14-mnemosyne-capture-stall]] (title + fifth read), + [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (115 → 210 slots), + [[incidents/2026-11-18-wiki-audit-credit-outage]] (→ resolved), [[systems/mnemosyne]], + [[current-state]], [[index]], `log.md`. +- **No job added, retired, disabled or repinned; no page removed or archived; no system obsoleted; no decision + reversed.** diff --git a/systems/skills-index.md b/systems/skills-index.md index ca50c2e..2be0947 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-20 -verified_on: 2026-11-20 +updated: 2026-11-21 +verified_on: 2026-11-21 confidence: high tags: [skills, hermes] sources: []