Compare commits

...
3 Commits
9 changed files with 806 additions and 1425 deletions
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident type: incident
status: resolved status: resolved
created: 2026-10-09 created: 2026-10-09
updated: 2026-10-24 updated: 2026-11-15
verified_on: 2026-10-24 verified_on: 2026-11-15
review_after: 2026-11-07 review_after: 2026-12-15
confidence: high confidence: high
tags: [cron, omniroute, router, timeout, incident, resolved] tags: [cron, omniroute, router, timeout, incident, resolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job) (`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again. closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
The storm contributes **no** unclosed rows. Nothing here is still symptomatic. The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened ## What happened
@@ -1,11 +1,11 @@
--- ---
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident type: incident
status: resolved status: degraded
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-25 updated: 2026-11-15
verified_on: 2026-10-25 verified_on: 2026-11-15
review_after: 2026-11-08 review_after: 2026-12-15
confidence: high confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23 # Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) **Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
**Status:** Resolved 2026-10-23 18:54 (patched and verified) **Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its **Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
memory layer. memory layer.
@@ -87,17 +88,39 @@ touching the file.
not positive proof that recall was populated on those calls. The proven part is that the call not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer. is no longer abandoned by the memory layer.
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
state at the 2026-11-15 03:00 stocktake:
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
(re-port + commit locally, or upstream it).
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
the *protection* is gone, not that the October fault returned.
## Follow-ups ## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json → - ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD** occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
(`0.64.0` → `0.69.0`). so "unexercised" is now moot.
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which - **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** → asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]]. [[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets - ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
(`kaishi00/hermes-conduit-notifier`).
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. - The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]] Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
+47 -12
View File
@@ -46,25 +46,60 @@ has no working_memory row (last capture 2026-11-14 01:32:59)
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`, - `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`. `first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`. Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and - ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
the page records no auth error text. **Observation, not verified diagnosis.** failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a - ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own ~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below. log line below.
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14 ## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the **An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
same hole and calling it benign: "already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
``` Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
```
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e. | line | content |
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The | --- | --- |
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning. | 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
**So the fault had ~12 hours of lead time that produced no alarm.** | 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
Two further corrections to the earlier revision:
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
sentence is withdrawn — 18:03 belongs to some other job.
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
(`-589.5 min` = 09:32:59 − 19:22).
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
the loudest signal available and the only one nothing watches.**
## Impact ## Impact
+4 -4
View File
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
confidence: high confidence: high
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
@@ -14,7 +14,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-11-15. > model routing and automation. Last updated: 2026-11-15.
> >
> **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **~12 h of lead time produced no alarm**: the 12:00:52 AWST run saw the same hole and logged it **ℹ️ informational** (`capture delay -589.5 min` — the negative sign hides the growth). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour). > **2026-11-14 (nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST) — a third TTS stack landed on CT200, and the memory pipeline went quiet.** **New page [[systems/phonon-tts]]**: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model **`gradium/phonon-40m-en`**, container created 2026-11-14 10:31:56 AWST), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`, 11:00:11), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`, 11:43:59) — verified live from CT460: `/health` 200, `/v1/models`, and a `POST /v1/audio/speech` returning a valid MP3; `/opt/ptts-voices` holds **7** voices; both ptts containers were restarted together at **17:54 AWST**. ⚠ **Provenance gap**: the retained transcripts cover only 08:35–09:03 AWST, so the 10:31 / 11:00 / 11:43 / 17:54 changes are **not attributable to any session in this profile** (working hypothesis: another estate agent — unproven). **New open incident [[incidents/2026-11-14-mnemosyne-capture-stall]]**: Mnemosyne has captured **nothing since 2026-11-14 09:32:59 AWST** — `working_memory` and `episodic_memory` stop in the same minute, last 8 working writes all 01:01–01:32 UTC — and the health monitor `ea79a2107e36` **escalated only at 2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault, state **`alerted`**, unclosed, alert text printing the raw **UTC** value so it reads as a 23 h gap when the true AWST gap was 14 h 31 m). ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 run saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run's own output block** (monitor log: header 957 = 12:01:52 run, header 1030 = 00:03:35 run, `-589.5` at 1046, ❌ at 1047). The earlier "~12 h of lead time" claim is **withdrawn**, as is the "exited 0 at 12:00 and 18:03" sentence (schedule `0 */12`; there was no 18:03 slot). Detection is **activity-aware** — `mnemosyne_freshness_gate.py` fires on `delay < −120 min` (magnitude-based, no sign-only test) and can only fire once a conversational message postdates the last capture; the conversation was idle 09:03→19:22, so midnight was the first slot able to see the miss. **What actually went unread: 29 consecutive 0-item consolidation cycles** from **10:04:42 AWST**, printed as `(0 items)` every run yet tested only for age (`> 36 h`). ~15 h of desktop (last message 19:22 AWST) / Telegram / cron context is uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established**. **Fleet otherwise clean — until the monitor's own 00:03 alert: 2026-11-14 settled 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit), deliveries **4 / 4 `delivered`**, no other incident row moved, so `cron_incidents` went **0 → 1 unclosed** (the monitor's row, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 (26 / 3)** — a **ninth** window unchanged, but **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. [[infrastructure/docker-services]] re-verified: CT200 **64 running non-seeder / 72 total** excluding the 1 transient seeder (was 61 / 68 on 11-07) — the delta is **exactly** the three TTS containers, **no removals**. The **2026-11-08 hook cascade** gained **no** new occurrence in the 11-14 body (the 00:05:40 timeout was already recorded by the 11-13 audit; `errors.log` 11-14 = 1 timeout + 10 suppression lines, otherwise only the benign `Browser Use viewing unavailable` plugin warning at ~2/hour).
> **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**. > **2026-11-13 (nightly audit; live probes 2026-11-13 23:59 - 2026-11-14 00:2x AWST) — the fleet went fully clean, then the hook came back.** **2026-11-13: 174 ledger rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:41) — the **third consecutive zero-`failed` day** (11-11 172/172, 11-12 176/176); deliveries **8 / 8 `delivered`**, 0 `for_failure`; `cron_incidents` **0 unclosed**. **Every job now reads `last_status: ok` with `failure_streak` 0 — the fleet has no non-clean row at all**, because **`066bb5191280` (Uber Eats Monthly Spend Report) was fixed end-to-end on 2026-11-13**, clearing a `delivery_failed` that had stood since 2026-11-01. Two stacked causes: the job's **runtime prompt had grown to 88 k chars** (it carried two heavy browser skills) so the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest, and the `delivery_failed` label was a **stale `TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart (**2026-11-13 13:14**). Fix: deterministic printer **`~/.hermes/cron/scripts/uber-eats-digest.py`**, rewritten prompt with a hard output contract, skills cleared (**88 k -> 1.8 k**); verified on the 18:51 -> 18:58 slot (**154 unique / 153 completed** orders, Oct **9 orders / $301.04**), lesson saved to `cron-job-maintenance`. Fleet **29 (26 / 3)** — an **eighth** window unchanged. ⚠ **But the window's fault is the `pre_tool_call` hook, and it returned:** `_pre_tool_call timed out after 120s` at **2026-11-14 00:05:40** plus **10** suppression lines through **00:06:38**, refusing this audit's own tool calls for ~1 minute (first-hand). `errors.log` by day: 11-12 **0/0**, 11-13 **0/0**, 11-14 **1/10** — the first such timeout in the retained log since the 2026-11-08 cascade (**`errors.log` now starts 2026-11-09 13:18:39**, so the 11-08 lines have rotated out and the page's 58-count stands on the 2026-11-09 read). **Load is the trigger and it has ~doubled: `load average 12.08 / 13.02 / 12.33` on 2 vCPU** (vs **5.89** at diagnosis), with **`state.db` now 2.04 GB** (**2,977 sessions / 207,577 messages**, was 1.89 GB / 2,901 / 197,682) and the root filesystem **88 % used**. The 30 -> 120 s fix is **proven insufficient**; [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] is **reopened, `resolved` -> `active`**, and `state.db` checkpoint/retention — not a bigger timeout — is now the estate's most urgent open item. Freshness (03:00 stocktake): **0 stale / 131 fresh / 3 no-date of 134**.
> >
> **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]]. > **2026-11-11 → 2026-11-12 (nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST) — a clean day, and an upgrade that could not be installed.** The 429 row that was the estate's only open incident **closed `resolved` at 2026-11-12T00:25:36** (the next slot of the same job ran clean, carrying both windows); *cron_incidents* now reads **0 unclosed of 137**. **2026-11-12: 176 ledger rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, started 23:59:54), deliveries **9 / 9 `delivered`**; **11-11: 172 / 172 `completed` / 0 `failed`**, deliveries 7 / 7. Fleet **29 (26 / 3)**, a seventh window unchanged. The window's one new open item is not a failure: the attempt to take **`hermes-hq-push`** past its installed **0.9.42** was **refused by the pre-install plugin security scan** — *"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."* The critical is the plugin's **own** `skills/dashboard-tile/SKILL.md` one-liner; **no per-verdict override exists** (`plugins.scan_on_install: false` would blind the scan for *every* future install, and `plugin_guard.py` hardcodes `trust_level="community"`). The fix is **PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file), pushed from a fork because `gh` is **read-only** upstream. Installed 0.9.42 keeps delivering. Also: the alleged *"stored hold-and-ask instruction"* for scan warnings **does not exist on disk** (184,716-file sweep, zero matches in editable text; the sentence lives only in `state.db` history) — nothing was removed or reverted, and the hunt is now the skill `stored-instruction-search-and-removal` (**skills 299 → 300**). `config.yaml` is **byte-identical to its 2026-11-12 18:14:52 backup**; the only drift since 11-09 is the **model catalogue block** (6 out, 9 in). **New page:** [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].
@@ -80,7 +80,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi
- [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. - [[systems/skills-index]] — Installed skills inventory (**300 skills** (**67 categories**) as of the **2026-11-13** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents ## Incidents
- [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — **mislabelled**, state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **The 12:00:52 AWST run saw the identical hole and logged it ℹ️ informational** (`capture delay -589.5 min`), i.e. **~12 h of lead time produced no alarm**. Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]]. - [[incidents/2026-11-14-mnemosyne-capture-stall]] - ⚠ **OPEN — Mnemosyne has captured nothing since 2026-11-14 09:32:59 AWST (~15 h at this read), and the health monitor took 14 h 31 m to say so.** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); the `pre_llm_call` capture hook logs `Mnemosyne capture ok` on every desktop turn while writing **0** rows, so the hook's own success is not evidence of capture. The monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth` — the scheduler's classifier, not a credential fault; this job's twelve earlier rows read `agent` / `timeout` / `unknown` — state **`alerted`**, unclosed), and its alert text prints the raw **UTC** instant, so it reads as a 23 h gap when the true AWST gap was 14 h 31 m. ⚠ **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 AWST run saw nothing** (`capture delay 29.4 min`, exit 0, `suppressed`) — the `-589.5 min` ℹ️ line belongs to the **00:03:35** run itself (monitor log lines 957 / 1030 / 1046). Detection is **activity-aware** (`mnemosyne_freshness_gate.py` fires on `delay < −120 min`, i.e. on **magnitude**) and the first message to postdate the stall arrived at **19:22 AWST**, after that slot — so the ~12 h of quiet was the `0 */12` **cadence**, not a threshold bug. **What went unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, visible as `(0 items)` in every run but tested only for age (36 h). Consequences: ~15 h of desktop / Telegram / cron context uncaptured, and the **23:30 sleep cycle consolidated 0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime **2026-11-13 18:03**; full YAML walk found no `[`/`{` scalars). Root cause **not established** → [[systems/mnemosyne]], [[systems/phonon-tts]].
- [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). - [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]] - **OPEN — the `hermes-hq-push` upgrade past v0.9.42 was refused by the pre-install plugin security scan.** `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` returned **"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict."** The sole critical is the plugin's **own** `skills/dashboard-tile/SKILL.md:41` shell one-liner (`echo '<json>' | python3 …` = *obfuscation*); the other 38 findings are low/medium and alone would have installed as `caution`. **`--force` lifts `caution` only**; a `dangerous` verdict is an unconditional block (`tools/plugin_guard.py:406-418`), and **no per-verdict override exists** — the only key is `plugins.scan_on_install` (default `true`, currently unset), which disables scanning for *every* future install; the `trust_level`/allowlist machinery lives in `tools/skills_guard.py` (the skills path) and `plugin_guard.py` hardcodes `trust_level="community"`. **Installed state unchanged:** v0.9.42, `enabled`, still delivering. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`, `fix/tile-skill-heredoc` → `main`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file, created 2026-11-12T10:00:01Z) — pushed from a fork because this host's `gh` account has **read-only** access upstream; local clone `/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
- [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]). - [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] - **RESOLVED 2026-11-12T00:25:36** (self-cleared on the next slot's clean completion; *cron_incidents* now 0 unclosed of 137) **— the auditor failed, so 2026-11-10 was never audited.** `cbfe434a3b5e` (Wiki Audit — Nightly) ran **2026-11-10T23:59:54.486572 -> 2026-11-11T00:12:29.646987 = 12 m 35 s** and died on `RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` (delivery `suppressed`) — **6 targets evaluated, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded. Incident row `cbfe43_ed01845310d8` (`rate_limit`) was raised at 00:12:28 and is **still open with `alerted_at: NULL` — the estate's only open incident row**. Fleet non-clean `last_status` **1 -> 2**; pool reset ≈ **06:36:57**. Third symptom of the same router behaviour (cf. [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]).
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]] - [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - ⚠ **REOPENED 2026-11-14 — the hook was the fault, and the 30 → 120 s fix did not hold at current load.** The `conduit_push` plugin is the **only** `pre_tool_call` registrant on the host (`__init__.py:119`) and its callback overran Hermes's **30 s** hook budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`errors.log`: **12** lines on 11-07, **46** on 11-08) plus **89** *"skipped after previous timeout or while still running"* suppression lines — every tool call degraded to slow-then-suppressed. **Fixed 2026-11-08 13:35 AWST: `plugins.hook_callback_timeout` 30 -> 120** in `~/.hermes/config.yaml`; the last timeout line remains 08:21:03 (re-verified at audit time). ⚠ **The plugin's caching was measured, not assumed, and was innocent** — `session_titles.py:21 CACHE_SECONDS = 30.0`, timed in-process at **3.1 s cold / 0.00 s warm**, so the *"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the 2-core CT460 (`load average 5.89`, `state.db` checkpointing a **40 MB WAL**). **Cost:** *Mnemosyne Health Monitor* **8 h 27 m** and **5 h 59 m 51 s**; *FreshRSS Midday* **2 h 09 m 53 s**; stocktake failed a third day. **Next lever is `state.db` checkpoint/retention, not more caching.** ⚠ **Recurred 2026-11-14 00:05:40** — **1 timeout + 10 skips**, overrunning the **120 s** budget at `load average 12.08` on 2 vCPU (**~2×** the 5.89 at diagnosis), with `state.db` grown to **2.04 GB**. Status **`resolved` → `active`**; the lever is now urgent. → [[systems/conduit]], [[systems/hermes-agent]]
+510 -1372
View File
File diff suppressed because one or more lines are too long
+149
View File
@@ -0,0 +1,149 @@
# Wiki freshness audit — 2026-11-15
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (133)
- [ 4d] CONTRIBUTING.md
- [ 7d] agents/hermes-chaos.md
- [ 7d] agents/hermes-production.md
- [ 4d] agents/index.md
- [ 17d] agents/nanobot.md
- [ 4d] agents/open-claw.md
- [ 11d] agents/qwenpaw.md
- [ 4d] agents/zeroclaw.md
- [ 17d] comparisons/camoufox-vs-chrome.md
- [ 4d] comparisons/hermes-vs-nanobot.md
- [ 4d] comparisons/memory-backends.md
- [ 4d] comparisons/model-routers.md
- [ 17d] comparisons/steel-vs-camoufox.md
- [ 4d] concepts/agent-memory.md
- [ 17d] concepts/browser-automation.md
- [ 4d] concepts/model-routing.md
- [ 4d] concepts/persistent-shells.md
- [ 4d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 17d] decisions/index.md
- [ 4d] decisions/keep-openrouter-as-fallback.md
- [ 4d] decisions/persistent-shell-enabled.md
- [ 17d] decisions/rollback-steel-to-camofox.md
- [ 4d] experiments/index.md
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 10d] incidents/2026-08-15-skills-index-drift.md
- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 4d] incidents/2026-08-29-gmail-read-state.md
- [ 3d] incidents/2026-09-02-sftpgo-outage.md
- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 20d] incidents/2026-10-03-ct450-hermes-update.md
- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 14d] incidents/2026-10-09-memory-recall-relevance.md
- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 16d] incidents/2026-10-29-conduit-voice-silence.md
- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 14d] incidents/2026-10-31-asana-access-failures.md
- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 0d] index.md
- [ 8d] infrastructure/backups.md
- [ 4d] infrastructure/credentials-map.md
- [ 8d] infrastructure/docker-services.md
- [ 4d] infrastructure/domains-and-tunnels.md
- [ 8d] infrastructure/hosts.md
- [ 8d] infrastructure/network-map.md
- [ 8d] infrastructure/proxmox.md
- [ 3d] infrastructure/tailscale.md
- [ 11d] queries/retained-research-answers.md
- [ 21d] reports/communities-wa-social-media-audit-2026.md
- [ 14d] reports/uber-eats/2026-07.md
- [ 14d] reports/uber-eats/2026-08.md
- [ 14d] reports/uber-eats/2026-09.md
- [ 2d] reports/uber-eats/2026-10.md
- [ 2d] reports/uber-eats/index.md
- [ 4d] runbooks/backup-wiki.md
- [ 4d] runbooks/diagnose-docker.md
- [ 4d] runbooks/gateway-resource-resilience.md
- [ 8d] runbooks/job-radar.md
- [ 4d] runbooks/multiplexer-setup.md
- [ 20d] runbooks/provider-health.md
- [ 17d] runbooks/recover-docker-service.md
- [ 4d] runbooks/restart-browser.md
- [ 4d] runbooks/restart-hermes.md
- [ 4d] runbooks/restore-openrouter.md
- [ 4d] runbooks/test-openrouter.md
- [ 4d] runbooks/troubleshoot-tool-loop.md
- [ 20d] runbooks/update-hermes-safely.md
- [ 7d] systems/browser-backend.md
- [ 4d] systems/cloudflare-tunnel.md
- [ 6d] systems/conduit.md
- [ 20d] systems/current-profile.md
- [ 4d] systems/guanaco.md
- [ 4d] systems/headless-server.md
- [ 6d] systems/hermes-agent.md
- [ 2d] systems/hermes-hq-push.md
- [ 4d] systems/hermes-ui.md
- [ 11d] systems/home-assistant.md
- [ 16d] systems/homepage-dashboard.md
- [ 11d] systems/ignis.md
- [ 20d] systems/memory-backend.md
- [ 13d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 20d] systems/model-providers.md
- [ 11d] systems/nanobot.md
- [ 4d] systems/nextcloud.md
- [ 4d] systems/obsidian.md
- [ 5d] systems/omniroute.md
- [ 4d] systems/opencode-go.md
- [ 4d] systems/openrouter.md
- [ 11d] systems/paseo.md
- [ 0d] systems/phonon-tts.md
- [ 5d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 4d] systems/terminal-backend.md
- [ 4d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages
+1 -1
View File
@@ -50,7 +50,7 @@ Primary agent memory store for Hermes.
## Known Issues ## Known Issues
**⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed) — but its **12:00:52** slot had already seen the identical hole and logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**; the threshold in the check script is on *sign*, not magnitude. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]] **⚠ OPEN — capture stalled at 2026-11-14 09:32:59 AWST and the monitor took 14 h 31 m to escalate (audit 2026-11-15 00:4x).** `working_memory` and `episodic_memory` stop in the same minute (last 8 working writes all 01:01–01:32 UTC); ~15 h of desktop / Telegram / cron turns are **not** in memory, and the 23:30 sleep cycle consolidated **0 items**. The health monitor `ea79a2107e36` exited 1 at **2026-11-15 00:03:44** on `STALE MEMORY PIPELINE` (incident `ea79a2_601037516abd`, `state: alerted`, unclosed; its `failure_type: auth` is the scheduler's classifier, not evidence of a credential fault). **Correction, re-derived 2026-11-15 01:2x: the 12:00:52 slot saw nothing** — it measured `capture delay 29.4 min` (healthy, exit 0, delivery `suppressed`), and the `-589.5 min` ℹ️ line belongs to the **00:03:35 run itself** (monitor log: header line 957 = 12:01:52 run, header line 1030 = 00:03:35 run, `-589.5` at line 1046, ❌ at 1047). `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`, default **120** min) but **activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last captured message was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min) and the first to postdate the stall was the **19:22 AWST desktop** turn — after the 12:00 slot. **The quiet was the `0 */12` cadence, not a sign-vs-magnitude bug.** **What did go unread was volume: 29 consecutive 0-item consolidation cycles from 10:04:42 AWST**, printed as `(0 items)` on every run yet tested only for age (`> 36 h`) — the candidate fix is a zero-item-streak threshold. ⚠ Related trap, verified: `consolidation_log.created_at` mixes formats (recent rows `+00:00`, older rows naive **AWST**), so treating the column as one axis misplaces events by 8 h. The known stringified-`config.yaml` killer is **ruled out** (`config.yaml` mtime 2026-11-13 18:03; full YAML walk shows no `[`/`{` scalars). Root cause **not established** — no error lines; the writes simply stop. Alert path verified working (delivery `for_failure: 1`, `delivered` 00:03:39). → [[incidents/2026-11-14-mnemosyne-capture-stall]]
**⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**. **⚠ Security - live `.env` values are stored verbatim in this store's own tables (live read 2026-11-06 00:2x-00:5x AWST).** **17 rows** carry three live credentials - `TELEGRAM_BOT_TOKEN` (4 rows), `MATTERMOST_TOKEN` (6) and `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` (7) - across `memories.content`, `working_memory.content`, `episodic_memory.content`, `gists.text`, `memoria_facts.context_snippet` and the `fts_episodes` / `fts_working` mirrors (per-key breakdown on the incident page). The 0-byte `episodic_memory.db` and `memoria_facts.db` stubs beside `mnemosyne.db` hold nothing - those names are *tables inside* the main DB. Capture stores verbatim session text (the 1.7 GB session store `state.db` itself carries all four values), so any session in which a key was printed becomes recall material. **Recall is the exposure path:** these rows can be pulled into any prompt, cron delivery or subagent transcript, and Mnemosyne's own rotation snapshots (seven uncompressed `.db` files under `backups/`) reproduce the whole DB. Not remediated. -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. Capture-time redaction of `.env` values is the durable fix; **rotation is the only remedy that also covers the existing copies**.
+44 -16
View File
@@ -24,11 +24,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`** error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the ("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at` `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5; its `failure_type: auth` is **not**
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver** evidence of a credential fault — this job's twelve earlier rows read `agent` ×3 / `timeout` ×1 /
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in `unknown` ×8, and the estate-wide table is 53 `unknown` / 47 `agent` / 15 `timeout`) →
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**. (`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — **notification
worked, and so did detection; the earlier slots simply had nothing to see.** *(Corrected 2026-11-15
01:2x — an earlier revision of this entry claimed the **12:00:52** slot had "already seen the identical
hole" and logged it ℹ️ at `-589.5 min`, giving "~12 h of lead time"; that was wrong on both counts. See
[[#Known Issues]] for the verified mechanism.)*
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor: **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed. **2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third - **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
@@ -311,17 +315,41 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
### Open 2026-11-15 ### Open 2026-11-15
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`, - **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled `failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`; the row's
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed … `failure_type: auth` is the scheduler's classifier and does **not** describe the fault — this job's
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true earlier rows read `agent` / `timeout` / `unknown`). Its 00:00 slot exited 1 at **00:03:44** on
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it `STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59` (raw **UTC** in the alert
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate text, so it *reads* as a ~23 h gap; the true AWST gap at the alert was **14 h 31 m**). The alert itself
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`). delivered (00:03:39, `for_failure: 1`). Root cause **not established**; the known
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page: stringified-`config.yaml` killer is ruled out. Full page:
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that [[incidents/2026-11-14-mnemosyne-capture-stall]].
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must - **Verified mechanism of the escalation (re-derivation pass, 2026-11-15 01:2x AWST — corrects the first
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing reading):** the **12:00:52** slot did **not** see the hole. `~/.hermes/logs/mnemosyne-health-monitor.log`
misreading trap. pins it by line number — run header `2026-11-14 12:01:52 AWST` at **line 957**, next header
(`2026-11-15 00:03:35 AWST`) at **line 1030**, and the `capture delay -589.5 min` line at **line 1046**,
i.e. *inside the 00:03:35 run's own block*, immediately above the ❌ at line 1047. The 12:00:52 run's own
measurement was `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay **29.4 min**` — healthy, exit 0,
delivery `suppressed`. `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`,
`MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min; consolidation age default **36 h**), but it is
**activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last
captured message before the stall was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min); the first
message to postdate it was the **19:22 AWST desktop** turn — *after* the 12:00 slot, so midnight was the
first slot that could see the miss. **The ~12 h of quiet was the `0 */12` cadence, not a threshold bug.**
- **What genuinely went unread — consolidation item counts.** The sleep cycles kept firing every ~31 min and
returned **0 items: 29 consecutive zero-item cycles**, streak opening `2026-11-14T02:04:42+00:00`
(**10:04:42 AWST**) and running to the alert; the last productive cycles were 4622/4621/4620 at
`09:33:34` / `09:33:16` / `09:01:40` (naive **AWST** — see the format trap below). Every run printed
`Last consolidation: … (0 items)`, including the healthy 12:00 one, but the gate tests consolidation
**age** (`> 36 h`) only — never the item count. A pipeline writing nothing still reads as fresh.
**Candidate threshold: an N-consecutive-zero-item-cycle exception.**
- **Format trap in `consolidation_log.created_at` (verified this pass):** rows 4640–4651 carry a `+00:00`
offset, rows 4620–4622 are **naive and local (AWST)**. Read as one axis they misplace events by 8 h and
make the streak look as if it began after the last productive cycle. Same family as the alert text below.
- **Lessons for this fleet:** (a) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
misreading trap (the alert's `01:32:59` is 09:32:59 AWST); (b) freshness gates must test a pipeline's
**output volume**, not just the age of its last run — an empty-but-current cycle is the failure signature
for a dead write path; (c) when a ledger's `failure_type` disagrees with the error text, trust the text —
the classifier is `unknown`/`agent` for most rows.
### Resolved 2026-11-02 ### Resolved 2026-11-02
+5 -6
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-14 updated: 2026-11-15
verified_on: 2026-11-14 verified_on: 2026-11-15
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []
@@ -12,9 +12,9 @@ sources: []
# Installed Skills # Installed Skills
**Total installed: 300 skills** in `~/.hermes/skills/`. **Total installed: 299 skills** in `~/.hermes/skills/`.
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
This page is auto-regenerated from the live skill inventory. Do not hand-edit. This page is auto-regenerated from the live skill inventory. Do not hand-edit.
@@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework - `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (108) ## Devops (107)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit - `agent-extension-audit` — # Agent Extension Audit
@@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `uber-eats-monitor` — # Uber Eats Order Monitor - `uber-eats-monitor` — # Uber Eats Order Monitor
- `vision-model-validation` — # Vision Model Validation via OmniRoute - `vision-model-validation` — # Vision Model Validation via OmniRoute
- `watchers` — # Watchers - `watchers` — # Watchers
- `web-content-extraction` — # Web Content Extraction — Fallback Techniques
- `weekly-comms-tracker` — # Weekly Communications Tracker - `weekly-comms-tracker` — # Weekly Communications Tracker
- `wiki-maintenance` — # Wiki Maintenance - `wiki-maintenance` — # Wiki Maintenance
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration - `zeroclaw-ops` — # ZeroClaw Operations & Configuration