wiki audit 2026-11-21: session-scoped capture re-diagnosis (124 Pulse rows, ~57 uncaptured turns, gateway-restart hypothesis NOT proven); NEW page — first content_policy_blocked digest failures (gmail 08eb86f02d37 + freshrss ecc1fc13655a); watchdog streak 115->210 (single incident row); wiki-audit credit outage RESOLVED (two nights lost, 11-20 delivery suppressed); ledger corrections (4 unclosed health-monitor rows, not 5; audit incident closed 2026-11-20T01:19:13; 7 unclosed cron_incidents); skills-index regenerated 299 skills/67 cats; freshness 5 stale/132 fresh/3 no-date; Quartz rebuilt+deployed (879 files)

This commit is contained in:
Tony0410
2026-11-21 00:40:00 +08:00
parent f603c29d17
commit ca0ff5aa80
11 changed files with 484 additions and 31 deletions
+15 -9
View File
@@ -3,16 +3,21 @@ title: Anthony's Systems Wiki
type: collection
status: active
created: 2026-07-22
updated: 2026-11-20
verified_on: 2026-11-20
updated: 2026-11-21
verified_on: 2026-11-21
confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout, content-policy, session-scoped-capture]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, live-sqlite-mnemosyne, raw/stocktake]
---
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. last updated: 2026-11-20
> model routing and automation. last updated: 2026-11-21
>
> **2026-11-21 (nightly audit; window 2026-11-20; live probes 2026-11-21 00:0x – 00:5x AWST) — the memory fault is session-scoped, a new provider content filter killed two digests, and the audit came back.**
> **⚠ Capture, corrected a second time.** The 11-20 finding that "**every** interactive session was captured" is **withdrawn** — that read counted *sessions*; capture is per *message*. Per message: the desktop DM session `20261031_190813_95fbde` owns **124** Pulse/automation rows in `working_memory` from **2026-10-31 11:09:16** to **2026-11-13 13:10:35 AWST** and **nothing after**, while the session kept producing turns — **~57** since, last captured turn **13:06:05**, first uncaptured **13:21:00** (verified in `state.db`). The **11-13 13:14 gateway restart** sits inside that 15-minute boundary — **leading hypothesis, explicitly NOT proven** (`agent.log`'s oldest surviving file begins **11-13 19:03:34**; the window has rotated out). Capture is **not** globally dead: the only write since, **`2026-11-18 19:25:03` AWST**, came from the new session `hermes_20261118_191808_5c9e08`. Counters frozen at `working_memory` **1,131** / `episodic_memory` **1,250**; **52 h 35 m** since the last write; the health monitor already says it — `STALE MEMORY PIPELINE: capture missed: user message`, `failure_streak: 4` → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]].
> **New failure class: `content_policy_blocked`.** Two unrelated digests died on the identical provider string on 11-20 — **Gmail Action Digest `08eb86f02d37` 08:31:06** and **FreshRSS Midday News Briefing `ecc1fc13655a` 12:01:14** — `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both `failure_streak: 1` (**isolated**, not chronic), both **alerted** and **unclosed**, both `failure_type: unknown` → [[incidents/2026-11-20-content-policy-blocked-digest-failures]].
> **The audit came back — and the outage was two nights, not one.** `cbfe434a3b5e` is `ok` / streak 0; the ledger dates it **11-18 00:00 failed · 11-19 00:00 failed · 11-20 00:00 completed** (report not delivered — `delivery_outcome: suppressed`) **· 11-21 00:00** this run. The 11-20 run absorbed windows **11-17/11-18** via its three-window catch-up; **no re-run was forced** → [[incidents/2026-11-18-wiki-audit-credit-outage]] now **`resolved`**. The **Mnemosyne Watchdog is still red — 210 consecutive 15-minute slots** (59 conflicts vs threshold 50), holding **one** incident id `96eb95_98ade422d195` from **11-18 20:01:15** to **11-21 00:31:16 (00:16:15 at the first read)** (**no fragmentation**), whereas the **health monitor** re-forms a fresh incident id every slot for one standing condition → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. Consolidation **0 items** (199 of the last 200 cycles). Fleet **29 jobs / 26 enabled**. Freshness **5 stale / 131 fresh / 3 no-date** of 139.
>
> **2026-11-20 (three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST) — the memory freeze broke by itself, the alarm did not, and the audits that would have said so had died twice.**
> **The capture stall is over — silently.** `working_memory` **1130 → 1131** / `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST**: **105 h 52 m**, ended with **no intervention**. And the "capture is dead" reading was wrong — `state.db` holds only **three** non-cron sessions in that window (two `desktop` on 11-14, both *before* the 09:33 boundary, then the 11-18 **19:18** session whose turn is the captured row), so **every interactive session was captured**; the alert measures **conversational silence** (age of the newest non-`cron`/`subagent` message). Corrected on [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]].
@@ -93,11 +98,12 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents
- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **OPEN — the nightly audit failed on two consecutive nights, so 2026-11-17 and 2026-11-18 were never audited, and no failure notice was ever sent.** `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`** — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]. The 2026-11-20 00:00 slot (this run) covers all three windows.
- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 115 consecutive watchdog failures from one cause: the conflict queue.** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 115`** at **2026-11-20 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 115 faults. Each slot delivers a *failure notice* (9 of the 24 deliveries since 11-18 are `for_failure: 1`), so the noisy channel is also the only channel; every slot re-opens incident `96eb95_98ade422d195` and nothing escalates or closes.
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (corrected 2026-11-20) — the freeze ended by itself at 2026-11-18 19:25 AWST after 105 h 52 m, and the "captured nothing" reading was a quiet estate, not a broken hook.** Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
- [[incidents/2026-11-20-content-policy-blocked-digest-failures]] - **OPEN — first `content_policy_blocked` failures on record: two unrelated digests killed by the identical provider string hours apart.** **Gmail Action Digest** `08eb86f02d37` ran **2026-11-20 08:31:06 → 08:32:45** and **FreshRSS Midday News Briefing** `ecc1fc13655a` ran from **12:01:14**, both ending `RuntimeError: content_policy_blocked: Output data may contain inappropriate content.` Both carry **`failure_streak: 1`** (isolated — each job's preceding slot completed), both **alerted** (incident rows `08eb86_c0cca1a55b96`, `ecc1fc_5a10b70830d1`) and both still **unclosed**, and both are classified **`failure_type: unknown`**, so the classifier cannot group the class or auto-close it. Nothing was changed on either job and **no re-run was forced**; whether the class recurs is unknown.
- [[incidents/2026-11-18-wiki-audit-credit-outage]] - **RESOLVED 2026-11-21 — the nightly audit failed on two consecutive nights, so windows 2026-11-17 and 2026-11-18 were never audited on their own night, and no failure notice was ever sent.** The ledger dates it exactly: `cbfe434a3b5e` failed **2026-11-18T00:00:08** and **2026-11-19T00:00:16** on a chained-provider **credit/auth** fault, then **completed on 11-20T00:00:08** (report not delivered — `delivery_outcome: suppressed`), absorbing both lost windows via its three-window catch-up; the job now reads `last_status: ok` / `failure_streak: 0`. **No re-run was forced**, and the silent-skip class it identified stays open on [[systems/scheduled-tasks]]. `cbfe434a3b5e` (Wiki Audit) ran **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02**, both closing `failed` on a chained-provider **credit/auth** fault (`failure_streak: 2`, last clean run 11-17). Both slots carry **`delivery_outcome: suppressed`**, so the estate's own auditor was the one thing nobody heard about; incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) was **never alerted** (`alerted_at: None`) — the same silent-skip shape as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] — and is now **resolved** (`closed_at` **2026-11-20 01:19:13**, i.e. it cleared itself on the first clean completion).
- [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] - **OPEN — 210 consecutive watchdog failures from one cause: the conflict queue (115 at the 11-20 read).** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **`failure_streak: 210`** at **2026-11-21 00:31:16** (`last_status: error`), because **59 unresolved conflicts sit above its threshold of 50** — a single durable trip, not 210 faults. Each slot delivers a *failure notice*, so the noisy channel is also the only channel. Unlike the health monitor, this job **de-duplicates**: **one** incident row `96eb95_98ade422d195` (`failure_type: script`) has carried all 210 slots — `first_seen` 2026-11-18 20:01:15, `last_seen` **2026-11-21 00:31:16**, `alerted_at` 2026-11-20 21:31:24, still **unclosed** — so nothing escalates and nothing closes.
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN (re-corrected 2026-11-21) — the 11-20 "quiet estate, not a broken hook" reading is WITHDRAWN.** Counted per *message* rather than per *session*, the desktop DM session `20261031_190813_95fbde` owns **124** `working_memory` rows ending **2026-11-13 13:10:35 AWST** with **~57** of its turns uncaptured since (last captured turn 13:06:05, first uncaptured 13:21:00; the **11-13 13:14 gateway restart** sits inside that gap — leading hypothesis, not proven). The **105 h 52 m** global freeze did end by itself at **2026-11-18 19:25 AWST** and capture still works for **new** sessions; the failing one is the long-lived desktop session. Earlier headline: **Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion — its incident row was closed 2026-11-12T00:25:36; *cron_incidents* stood at 137 rows at that read) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and was **never alerted** (`alerted_at: NULL`) and is now **resolved** (`closed_at` **2026-11-12T00:25:36**) — so "the estate's only open incident row" has been overtaken: unclosed *cron_incidents* at the **2026-11-21** read is **7** (`08eb86` ×1, `ecc1fc` ×1, `ea79a2` ×4, `96eb95` ×1). Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]] (resolved 2026-11-11) - **RESOLVED BY ITSELF.** The nightly stocktake closed `completed` after **9 h 10 m** having landed **zero edits and no commit** — `failure_streak` 3 → 0 while the work stayed undone, and the top-priority item on its brief had been the monotonically worsening failure streak. Third consecutive day its findings never landed.
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]] - **NO INCIDENT RAISED.** Job Radar Stage 1 overran to **9 h 09 m**, skipped its own verification, wrote **no handoff file**, and both Stage-2 slots ran before it finished and declined silently. No incident row was created — the completion-time-only alerting gap leaves the failure invisible.