Compare commits

...
9 Commits
Author SHA1 Message Date
Tony0410 f603c29d17 stocktake 2026-11-20: re-verify 9-page 2026-10-29 stale cohort live; correct nanobot docker/cron drift, camofox host/port claims, dead-pin residual; freshness 0 stale/136 fresh 2026-11-20 03:11:58 +08:00
Tony0410 8bf0a37022 wiki audit 2026-11-20 (publish): skills-index regenerated (299 skills/67 categories), freshness report 20261120 (9 stale/127 fresh/3 no-date), log entry extended with freshness cohort + publish verification 2026-11-20 01:13:20 +08:00
Tony0410 226767f3ec wiki audit 2026-11-20: three-window catch-up (11-17/11-18/11-19, both prior audit slots died)
- new: incidents/2026-11-18-wiki-audit-credit-outage (audit failed 11-18 + 11-19, deliveries suppressed, incident row alerted_at=None)
- new: incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm (115 consecutive slots, 59 unresolved conflicts vs threshold 50)
- corrected: incidents/2026-11-14-mnemosyne-capture-stall + systems/mnemosyne — freeze ENDED 2026-11-18 19:25 AWST (105h52m, no intervention); 'capture dead' reading was conversational silence (every interactive session captured)
- systems/scheduled-tasks: new Count bullet 11-17 173(171/2) / 11-18 172(153/19) / 11-19 176(78/98); fleet 29 (26/3); non-clean last_status 1->4; unclosed incidents 5->4; deliveries 24 (23 delivered / 1 failed)
- index.md feed + incidents + systems lines; log.md dated entry; current-state.md new section; frontmatter dates -> 2026-11-20
2026-11-20 01:02:44 +08:00
Tony0410 16f2c878c0 wiki stocktake 2026-11-19: 3 stale pages re-verified (conduit-call open item re-probed, 10-26 orphaned-run incident resolved, WA communities report live-checked); freshness report; skills-index dates 2026-11-19 03:22:49 +08:00
Tony0410 98610466cc 2026-11-18 stocktake: re-verify 7 stale pages against live state, resolve gmail-digest refire incident 2026-11-18 03:19:54 +08:00
Tony0410 4574c4ebc3 wiki-maintain 2026-11-17: regenerate skills-index (299 skills/67 cats), freshness report (7 stale); verified live & updated current-profile (ops profile deleted, config v52, 6 MCP servers, 15 plugins, 21/26 pins), memory-backend (2901 memories, coverage working/episodic 100%), model-providers (pin split, pulse_memory aux slot on dead nous credential) 2026-11-17 03:11:54 +08:00
hermes e5612c8a8d audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11 2026-11-17 00:36:27 +08:00
Tony0410 4eee02b9a2 wiki-maintain 2026-11-16: regenerate skills-index (299 skills/67 cats), freshness report (1 stale: communities-wa audit, review_after 2026-11-26) 2026-11-16 03:03:34 +08:00
Tony0410 9f80926dda audit 2026-11-15: capture stall 38h46m (freeze total, counts unchanged); cron_incidents 1->3 unclosed (signature embeds elapsed-hours float); monitor's 4.64h decoded as newest-conversational-message age; watchdog lost an 11:45 slot to a 600s deps-guard timeout; ledger 168 rows 165/3 (11-14 settled 167/0) 2026-11-16 00:32:12 +08:00
39 changed files with 1750 additions and 115 deletions
+40 -15
View File
@@ -3,8 +3,8 @@ title: Nanobot
type: agent
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [agent, nanobot, homelab]
sources: [live-probe, runtime-config]
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
- **Workspace:** `/home/nano/.nanobot/workspace`
## Current state
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening.
- **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
@@ -38,22 +38,31 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
`gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`.
~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
## Cron jobs
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded:
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST
- `morning-quota-report` — daily at 07:30 AWST
- `daily-quota-730am` — daily at 07:30 AWST
- `vault-daily-seed` — daily at 08:00 AWST
- `daily-agent-briefing` — daily at 08:15 AWST
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST
- `daily-quota-7pm` — daily at 19:00 AWST
- `evening-quota-report` — daily at 21:00 AWST
- `heartbeat` — every 30m
- `dream` — every 2h (memory consolidation)
**Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
(all schedules Australia/Perth):
| Job | Schedule | Enabled |
|---|---|---|
| `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
| `nightly-skills-export-sync` | daily 04:00 | ✅ |
| `morning-news-podcast` | daily 07:00 | ✅ |
| `daily-quota-730am` | daily 07:30 | ✅ |
| `morning-quota-report` | daily 07:30 | ❌ disabled |
| `vault-daily-seed` | daily 08:00 | ✅ |
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
| `daily-quota-7pm` | daily 19:00 | ✅ |
| `evening-quota-report` | daily 21:00 | ❌ disabled |
| `heartbeat` | every 30m | ✅ |
| `dream` | every 2h (memory consolidation) | ✅ |
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
## Skills
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
@@ -73,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
- [[runbooks/restart-hermes]]
- [[runbooks/nanobot-restart]]
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
> agentmail-daemon and readlater (:3210) processes all still run.
>
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
>
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
## Related
- [[comparisons/hermes-vs-nanobot]]
+12 -2
View File
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
type: comparison
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: medium
tags: [comparison, browser]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -46,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
> No Chrome/Chromium process found in this probe either.
### Chrome (not found in use)
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
+9 -2
View File
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
type: comparison
status: active
created: 2026-07-29
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [comparison, browser, steel, camofox, decision]
sources: [live-probe, session-history]
@@ -77,6 +77,13 @@ the CT201 and CT333 deployments torn down — so returning to Steel means redepl
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
> `cloud_provider: camofox`. All removal claims **hold**.
## Related
- [[systems/browser-backend]]
- [[concepts/browser-automation]]
+6 -2
View File
@@ -3,8 +3,8 @@ title: Browser Automation
type: concept
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [concept, browser, camofox, steel]
sources: [live-probe, session-history]
@@ -33,6 +33,10 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
## Related
- [[systems/browser-backend]]
- [[comparisons/steel-vs-camoufox]]
+77 -4
View File
@@ -3,16 +3,89 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
---
# Current State
Updated: 2026-11-15
Updated: 2026-11-20
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
**Previous window:**
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
load-timeout storm nobody had logged.**
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
streak **120** cycles. No cause, no fix, `status: active` →
[[incidents/2026-11-14-mnemosyne-capture-stall]]
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
cleared with no fix applied. The companion
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
a *larger* `state.db`).
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
all three alerts read alike. **Do not read it as the stall length.**
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
human-side for the wiki to have missed.
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
no-date of 136** pages.
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
+6 -2
View File
@@ -3,8 +3,8 @@ title: Decisions
type: collection
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [decisions]
---
@@ -31,6 +31,10 @@ Stable strategic and config decisions, with evidence and status.
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
> backend.
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
## History
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
+11 -3
View File
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
type: decision
status: active
created: 2026-08-02
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [decision, browser, steel, camofox, rollback]
sources: [session-history, live-config]
@@ -50,7 +50,7 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
> restoration was needed or wanted.
## Rationale
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
@@ -103,6 +103,14 @@ removed**, so the `plugin_steel` toolset is no longer selectable.
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
> The decision and its consequences are unchanged.
## Related
- [[comparisons/steel-vs-camoufox]]
- [[systems/browser-backend]]
@@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident
status: resolved
created: 2026-10-02
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
@@ -130,6 +130,12 @@ done.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog.
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
## Related
- [[incidents/2026-10-05-dead-model-pin-repin]]
+6 -3
View File
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [hermes, ct450, chaos, update, incident]
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
@@ -49,7 +49,10 @@ Two gotchas that cost time:
- CT460 production untouched — separate config, skills, cron and memory state.
## Current state (re-verified 2026-10-26)
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
## Current state (re-verified 2026-11-18)
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
## Follow-ups
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
type: incident
status: resolved
created: 2026-10-04
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
confidence: high
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
@@ -65,6 +65,9 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
figures below are superseded.
## Current state (re-verified 2026-11-18)
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
## Follow-ups
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
+13 -2
View File
@@ -3,8 +3,8 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident
status: resolved
created: 2026-10-06
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
@@ -119,6 +119,17 @@ state rather than left on trust.
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
> [[systems/scheduled-tasks]], not here.
## Related
- [[systems/scheduled-tasks]]
@@ -3,8 +3,8 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [cron, router, provider, infrastructure]
@@ -35,6 +35,8 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
> shut, the single-node exposure is not.
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
@@ -3,8 +3,8 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
type: incident
status: active
created: 2026-10-21
updated: 2026-10-27
verified_on: 2026-10-27
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-03
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
| ~09:10 | the orphaned execution row closes as **`unknown`** |
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
## Impact
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
@@ -3,8 +3,8 @@ title: Conduit Call Requests Refused by the Agent
type: incident
status: open
created: 2026-10-25
updated: 2026-10-26
verified_on: 2026-10-25
updated: 2026-11-19
verified_on: 2026-11-19
review_after: 2026-11-08
confidence: high
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
@@ -93,3 +93,11 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
- **Status stays open**: the request path is untested, not confirmed good.
## Status update — 2026-11-19 (stocktake re-probe)
- `~/.hermes/conduit-calls.json` → `history` is **still empty (0 entries)** as of 2026-11-19 03:0x
AWST — the call path has now gone unused for ~26 days since the fault. The "verify a real call
end-to-end" open item remains outstanding.
- No new refusal episodes have been recorded against this page. **Status stays open** on the same
basis as before: untested, not confirmed good.
@@ -1,13 +1,13 @@
---
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
type: incident
status: open
status: resolved
created: 2026-10-26
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
review_after: 2026-11-09
confidence: medium
tags: [cron, automation, hermes, incident, unresolved]
tags: [cron, automation, hermes, incident]
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
---
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
execution row.
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
## Resolution (verified 2026-11-18)
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
@@ -1,11 +1,11 @@
---
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
type: incident
status: active
status: resolved
created: 2026-10-27
updated: 2026-10-28
verified_on: 2026-10-28
review_after: 2026-11-03
updated: 2026-11-19
verified_on: 2026-11-19
review_after: 2026-12-19
confidence: high
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
@@ -111,3 +111,15 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
(`timed_out=True` at 1.35 s of a 30 s budget).
## Update — 2026-11-19 (stocktake re-probe)
- **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3`
`resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13**
(live probe of `cron_incidents` in `~/.hermes/cron/executions.db`).
- The `lost its durable fire claim ownership` error string has **no rows in the retained
`executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention
floor; the 10-28 update already confirmed the 10-27 window clean).
- **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups
1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics)
remain open design questions and were **not** actioned.
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
type: incident
status: active
created: 2026-11-09
updated: 2026-11-14
verified_on: 2026-11-14
review_after: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-20
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
captured none of this**. Exact `errors.log` counts for 11-15:
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
rather than a mechanism.
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
## See also
- [[systems/conduit]]
+197 -8
View File
@@ -1,17 +1,21 @@
---
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
type: incident
status: active
created: 2026-11-15
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-17
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
---
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
> not a broken hook. Read the newest re-read section before the earlier readings.
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
@@ -21,6 +25,181 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
the same condition was classified informational — see below).
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
then both stopped inside the same minute.
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
observation, no `last_seen_at` accumulation, no escalation:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
call): normalise the value out of the signature — round it, or omit it — so the same condition
**re-opens the existing row** instead of fragmenting.
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
*conversational* message — not the age of the newest capture.**
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
| --- | --- | --- | --- |
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
that is not one.
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
captured before the stall — the 2026-11-14 00:09:12 run logged
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
### The consolidator is alive and every cycle is a deliberate no-op
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
what would have caught this at 10:04 on 11-14.**
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
```
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
```
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
times out at 600 s belongs next to this stall rather than dismissed.
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
total.**
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
**62 h 54 m** with **zero** writes.
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
| Incident id | `first_seen_at` | `alerted_at` |
| --- | --- | --- |
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
on any single row.
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
reason the stall cannot self-heal.
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
| --- | --- | --- |
| `working_memory` | 1130 | **1131** |
| `episodic_memory` | 1249 | **1250** |
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
a broken capture path — is the durable defect on this page.
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
applied**.
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
@@ -103,6 +282,11 @@ the loudest signal available and the only one nothing watches.**
## Impact
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
window. Recall over that period will answer from nothing.
@@ -133,7 +317,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
## Related
@@ -0,0 +1,117 @@
---
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
type: incident
status: unresolved
created: 2026-11-17
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-11-24
confidence: high
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
---
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
`errors.log` grep, not an estimate.
## What the log records (2026-11-15)
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
older lines):
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
| --- | --- | --- | --- | --- | --- |
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
### By plugin (2026-11-15)
| Plugin | Failed loads | Registrations dropped |
| --- | --- | --- |
| `hermes-hq-push` | 26 | 0 |
| `conduit_push` | 9 | **474** |
| `homeassistant` | 7 | 34 |
| `browser-browser-use` | 3 | 3 |
| `basic` | 2 | 2 |
| `cadu-secrets-vault` | 1 | 4 |
| `telegram-platform` | 1 | 1 |
| `nous` | 1 | 0 |
| `drain` | 1 | 0 |
| `browser-browserbase` | 1 | 1 |
| **Total** | **52** | **519** |
## The dropped registrations are the more serious half
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
[[systems/conduit]].
## Clustering
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
spread over hours 01, 04–11 and 13:
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
are refused in the same session (`local-989b3642…`).
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
dropping ~52–56 registrations.
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
failure alongside it.
## What this is — and is not
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
**saturation symptom** on this host, not a wrong setting.
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
guard also blew its **600 s** budget that day (see
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
exceeded in the same 24 h.
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
predict it**.
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
## Open question
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
it is Anthony's call).
## See also
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
- [[systems/hermes-agent]] — plugin loading, config and timeouts
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
@@ -0,0 +1,111 @@
---
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
---
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
re-reported every 15 minutes, not a series of separate faults.
## The failure is a fixed, self-describing threshold trip
```
Script exited with code 1
stdout:
🚨 Many unresolved conflicts: 59 (threshold: 50)
```
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
re-alerts forever.
## Alerting works — and that is the problem
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
| delivered | execution |
|---|---|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
arriving on the same job is indistinguishable from the standing one.
## What the 59 conflicts actually are
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
unresolved.** Unresolved by creation day:
| created | unresolved |
|---|---|
| 2026-10-31 | 11 |
| 2026-11-02 | 2 |
| 2026-11-03 | 12 |
| 2026-11-06 | 1 |
| 2026-11-07 | 2 |
| 2026-11-08 | 1 |
| 2026-11-12 | 13 |
| 2026-11-13 | 3 |
| **2026-11-18** | **14** |
| **total** | **59** |
Two things stand out:
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
applied here**.
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
not opened in this window.
## The 2026-11-18 batch coincides with the migration session
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
the fact pairs were not inspected.
## Not applied / open
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
- The watchdog's cadence, signature construction and alert path were left as-is.
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
Same family as the zero-item-consolidation streak problem on
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
the gate tests a level, not a change.
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
@@ -0,0 +1,89 @@
---
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
type: incident
status: active
created: 2026-11-20
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
---
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
reads `running` (started `2026-11-20T00:00:08`).
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
| slot | started | finished / last seen | duration | status |
|---|---|---|---|---|
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
```
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
make this request. Please purchase more credits to continue using the service. (HTTP 400);
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
openai-compati…
```
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
So the outage produced **no push to Anthony** and the red cron row was the only signal.
## Why this matters beyond one job
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
## Precedent — and a repeating pattern
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
rather than at the alert throttle.
## Not applied / open
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
cron entry was edited.
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
entry appended to `log.md` the same night).
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
basis: the truncated 314-char error string.
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
+21 -6
View File
File diff suppressed because one or more lines are too long
+108
View File
@@ -1782,3 +1782,111 @@ provider, so the cold-start path is untested. Page → `status: degraded` with a
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST)
**Verdict: 1 new page + 7 page updates, nothing archived, no new system.**
**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173
`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`);
deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15
watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load
6.19–7.48. **No new system, no new concept, no page worth archiving.**
**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous
audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on
**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded
`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools,
skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in
01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0
/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is
a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`).
**Pages:**
- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the
load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the
open question (fail-open loader; a registration health check is proposed, **not applied**).
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for
the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak
**5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**.
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in
two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire
**without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor.
- [[systems/conduit]] — 474 dropped registrations and what they cost.
- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers.
- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up).
- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped.
**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold
(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and
recorded here so the next audit does not re-litigate it.
**Still open from earlier windows:** the capture stall has no established cause and no fix applied
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.
## 2026-11-18 03:xx — 03:00 stocktake (cron)
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged vs 11-17), freshness report `raw/stocktake/freshness-20261118.txt` — 9 stale.
- Live-vs-page re-verification of stale pages; 7 re-verified with dated corrections:
- `runbooks/provider-health.md` — snippets re-executed: CT600 /models 200 (6,501), omni-101 /api/health 200, fallbacks 200.
- `runbooks/update-hermes-safely.md` — prerequisites re-verified (v0.23.0, venv + managed runtimes present).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 now v0.23.1 (0 behind origin/main), CT460 v0.23.0; CT255 mattermost running + peer online → Mattermost-timeout claim superseded.
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — coverage monitor ok/streak 0; fix holds (script blocked by lifecycle guard; verified via monitor).
- `incidents/2026-10-25-gmail-digest-runaway-and-refire.md` — RESOLVED: last_status ok, one clean daily run 11-12→11-17, no refires; delivery_queued cleared.
- `incidents/2026-10-16-omniroute-503-resource-pressure.md` — 22 days clean, stays monitoring.
- `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` — no unknown rows in retained window; 10-20 log line aged out (rotation noted on page).
- Left stale (flagged only, not verifiable this run): `reports/communities-wa-social-media-audit-2026.md` (24d), `incidents/2026-10-24-conduit-call-request-refused.md` (23d, open behavioural).
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
## 2026-11-19 03:xx — 03:00 stocktake (cron)
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged), freshness report `raw/stocktake/freshness-20261119.txt` — 3 stale.
- All 3 stale pages re-verified against live sources with dated corrections:
- `incidents/2026-10-24-conduit-call-request-refused.md` — `conduit-calls.json history` still 0 entries (call path unexercised ~26d); status stays open.
- `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions.
- `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements.
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
## 2026-11-20 00:5x — nightly audit (three windows: 2026-11-17, 11-18 and 11-19 AWST)
**Why three windows:** the audit's own slots at **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02** both closed `failed`, so **2026-11-17 and 2026-11-18 were never audited**. This run covers all three.
- **New page — [[incidents/2026-11-18-wiki-audit-credit-outage]]:** `cbfe434a3b5e` (Wiki Audit — Nightly) is at `failure_streak: 2`, both failures on a chained-provider **credit/auth** fault (last clean run 11-17). Both failed slots carry `delivery_outcome: suppressed`, so **no failure notice ever reached Anthony** — the estate's own auditor was the one thing nobody heard about. Its incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`**, the same silent-skip shape as the 11-10 429 incident.
- **New page — [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]:** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **115 consecutive failures** at **2026-11-20 00:31:16** (`failure_streak: 115`, `last_status: error`), for one reason: **59 unresolved conflicts against its threshold of 50**. A single durable trip, not 115 faults — but each slot delivers a fresh *failure notice* (9 of the 24 deliveries since 11-18 carry `for_failure: 1`), so the noisy channel is the estate's only channel, and every slot re-opens incident `96eb95_98ade422d195`.
- **Corrected — [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]:** the freeze is **over**. `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** → the stall ran **105 h 52 m** and ended **with no intervention** (no config edit, restart or repair on the capture path). ⚠ **And the "capture is dead" reading was wrong:** `state.db` lists only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` measures the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect, and the monitor is still failing on it (`failure_streak: 2`; 11-19 00:01 was the only clean slot). Title and body reframed accordingly.
- **[[systems/scheduled-tasks]] — new Count bullet (11-17 / 11-18 / 11-19):** ledger **173 rows (171 `completed` / 2 `failed`)** / **172 (153/19)** / **176 (78/98)** — the last is the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18: **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (a *content* delivery, `live adapter send failed: Timed out`, `2026-11-19 08:34:13`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window** at the same count (11-18 and 11-19 went unrecorded, see above) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` `error` (2), `96eb95ce12f2` `error` (115), `ea79a2107e36` `error` (2), `08eb86f02d37` `delivery_failed` (0). `cron_incidents` unclosed **5 → 4**. The 03:00 stocktake (`1a8e90ea2271`) ran `completed` on all three days — **11 m 35 s / 23 m 07 s / 24 m 38 s** — so the estate's structural maintenance kept running while both alarm paths were red.
- **[[current-state]]** — new Recent Changes section for the 2026-11-17 → 2026-11-20 window. Frontmatter `updated` / `verified_on` bumped to **2026-11-20** across `index.md`, `current-state.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` and the two touched incident pages.
- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**.
- **Freshness at publish time** (`raw/stocktake/freshness-20261120.txt`): **9 stale / 127 fresh / 3 no-date of 139 pages**. All nine carry the *same* `updated: 2026-10-29` — one 22-day bulk cohort, not nine findings: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. **Not re-verified by this audit** — the live-vs-page re-verification sweep belongs to the 03:00 stocktake (pattern 23), so the cohort is recorded here rather than silently inherited. Five further pages sit at exactly 21d and become the 11-21 stale list: `incidents/2026-10-27-job-radar-budget-exhausted.md`, `incidents/2026-10-28-job-radar-stage1-iteration-cap.md`, `incidents/2026-10-29-conduit-voice-silence.md`, `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md`, `systems/homepage-dashboard.md`.
- **Published (patterns 13/30):** `wiki-maintain.sh` regenerated `systems/skills-index.md` (**299 skills / 67 categories**) and rebuilt Quartz — 139 pages parsed, 867 files emitted to `public/`, `hermes-wiki-serve.service` restarted (`systemctl --user is-active` → `active`, listener on `:9120`). Live check after the rebuild: `index`, `current-state`, `systems/mnemosyne`, `systems/scheduled-tasks`, `systems/skills-index` and all three incident pages return **200** on `http://127.0.0.1:9120/<path>`. Both vault conventions synced from the authoring tree with per-file `sha256` matches (8 files × 2 conventions; freshness report included), committed locally — the vault repo has no remote.
## 2026-11-20 03:0x — Stocktake (cron 1a8e90ea2271): 9-page stale cohort re-verified live
- Ran `wiki-maintain.sh` (repo script): skills-index regenerated (**299 skills / 67 categories**), freshness report `raw/stocktake/freshness-20261120.txt` → **9 stale / 127 fresh / 3 no-date of 139**. All nine stale pages were the single 2026-10-29 cohort (22d) flagged by the 01:13 audit — treated as one verification sweep (pattern 23), not nine investigations.
- **Live probes:** `~/.hermes/config.yaml` browser block; `ss -ltnp` + process list + `cua-driver --version` on CT460; `camofox-browser.service` (`systemctl --user` → active) + `:9377/health` → ok; CT333 via PVE (`ss`, `ps`, `docker ps -a`, `tailscale ip -4`, gateway `/health`, `workspace/cron/jobs.json`); CT201 via PVE (`docker ps -a`, images, Serve status, `_archived-steel/`, `:8085` probe); OmniRoute chat-completions on both nodes (`cliprox/terra`, `free-stack`, `smart-route`); `command-code/gpt-5.6-terra`; `jobs.json` fleet read.
- **All 9 pages re-verified** (`updated`/`verified_on` → 2026-11-20) with dated notes: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`.
- **Corrections (claims disproved or drifted):**
- `agents/nanobot.md`: Docker list drifted — `tender_brattain` → `ecstatic_swartz` (random-name rename class), `quirky_darwin` removed (absent with `-a`), stopped `qdrant` present. Gateway PID now 1256104 (v0.4.1, `/health` ok). **Cron section rewritten from live scheduler**: 11 jobs / 8 enabled; `morning-quota-report`, `daily-agent-briefing`, `evening-quota-report` disabled; `rss-check-proxmox-selfhst` gone; new `nightly-skills-export-sync` (04:00) and `morning-news-podcast` (07:00). Quota pair 07:30/19:00 remains enabled. Scheduler file is `workspace/cron/jobs.json` (the `~/.nanobot/cron/` dir is empty).
- `comparisons/camoufox-vs-chrome.md`: struck the stale "9377 not listening on CT460 / service on CT450" claim — `:9377` **is** listening on CT460 (local `camofox-browser.service`, active); ws-server also on `:9222`.
- `decisions/rollback-steel-to-camofox.md`: Rationale's "(CT450)" host attribution corrected to **CT460** with strikethrough.
- `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27).
- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists).
- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced.
+149
View File
@@ -0,0 +1,149 @@
# Wiki freshness audit — 2026-11-16
Threshold: >21 days since `updated`
## STALE (1)
- [ 22d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
## FRESH (132)
- [ 5d] CONTRIBUTING.md
- [ 8d] agents/hermes-chaos.md
- [ 8d] agents/hermes-production.md
- [ 5d] agents/index.md
- [ 18d] agents/nanobot.md
- [ 5d] agents/open-claw.md
- [ 12d] agents/qwenpaw.md
- [ 5d] agents/zeroclaw.md
- [ 18d] comparisons/camoufox-vs-chrome.md
- [ 5d] comparisons/hermes-vs-nanobot.md
- [ 5d] comparisons/memory-backends.md
- [ 5d] comparisons/model-routers.md
- [ 18d] comparisons/steel-vs-camoufox.md
- [ 5d] concepts/agent-memory.md
- [ 18d] concepts/browser-automation.md
- [ 5d] concepts/model-routing.md
- [ 5d] concepts/persistent-shells.md
- [ 5d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 18d] decisions/index.md
- [ 5d] decisions/keep-openrouter-as-fallback.md
- [ 5d] decisions/persistent-shell-enabled.md
- [ 18d] decisions/rollback-steel-to-camofox.md
- [ 5d] experiments/index.md
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
- [ 5d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 11d] incidents/2026-08-15-skills-index-drift.md
- [ 12d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 4d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 5d] incidents/2026-08-29-gmail-read-state.md
- [ 4d] incidents/2026-09-02-sftpgo-outage.md
- [ 12d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 18d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 21d] incidents/2026-10-03-ct450-hermes-update.md
- [ 21d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 18d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 1d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 15d] incidents/2026-10-09-memory-recall-relevance.md
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 20d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 6d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 20d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 21d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 21d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 19d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 17d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 17d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 17d] incidents/2026-10-29-conduit-voice-silence.md
- [ 16d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 17d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 16d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 15d] incidents/2026-10-31-asana-access-failures.md
- [ 15d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 15d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 14d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 13d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 12d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 10d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 10d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 11d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 7d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 11d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 9d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 7d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 5d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 5d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 4d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 3d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 0d] index.md
- [ 9d] infrastructure/backups.md
- [ 5d] infrastructure/credentials-map.md
- [ 9d] infrastructure/docker-services.md
- [ 5d] infrastructure/domains-and-tunnels.md
- [ 9d] infrastructure/hosts.md
- [ 9d] infrastructure/network-map.md
- [ 9d] infrastructure/proxmox.md
- [ 4d] infrastructure/tailscale.md
- [ 12d] queries/retained-research-answers.md
- [ 15d] reports/uber-eats/2026-07.md
- [ 15d] reports/uber-eats/2026-08.md
- [ 15d] reports/uber-eats/2026-09.md
- [ 3d] reports/uber-eats/2026-10.md
- [ 3d] reports/uber-eats/index.md
- [ 5d] runbooks/backup-wiki.md
- [ 5d] runbooks/diagnose-docker.md
- [ 5d] runbooks/gateway-resource-resilience.md
- [ 9d] runbooks/job-radar.md
- [ 5d] runbooks/multiplexer-setup.md
- [ 21d] runbooks/provider-health.md
- [ 18d] runbooks/recover-docker-service.md
- [ 5d] runbooks/restart-browser.md
- [ 5d] runbooks/restart-hermes.md
- [ 5d] runbooks/restore-openrouter.md
- [ 5d] runbooks/test-openrouter.md
- [ 5d] runbooks/troubleshoot-tool-loop.md
- [ 21d] runbooks/update-hermes-safely.md
- [ 8d] systems/browser-backend.md
- [ 5d] systems/cloudflare-tunnel.md
- [ 7d] systems/conduit.md
- [ 21d] systems/current-profile.md
- [ 5d] systems/guanaco.md
- [ 5d] systems/headless-server.md
- [ 7d] systems/hermes-agent.md
- [ 3d] systems/hermes-hq-push.md
- [ 5d] systems/hermes-ui.md
- [ 12d] systems/home-assistant.md
- [ 17d] systems/homepage-dashboard.md
- [ 12d] systems/ignis.md
- [ 21d] systems/memory-backend.md
- [ 14d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 21d] systems/model-providers.md
- [ 12d] systems/nanobot.md
- [ 5d] systems/nextcloud.md
- [ 5d] systems/obsidian.md
- [ 6d] systems/omniroute.md
- [ 5d] systems/opencode-go.md
- [ 5d] systems/openrouter.md
- [ 12d] systems/paseo.md
- [ 1d] systems/phonon-tts.md
- [ 6d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 5d] systems/terminal-backend.md
- [ 5d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 1 stale, 132 fresh, 3 no-date of 136 pages
+150
View File
@@ -0,0 +1,150 @@
# Wiki freshness audit — 2026-11-17
Threshold: >21 days since `updated`
## STALE (7)
- [ 23d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
- [ 22d] incidents/2026-10-03-ct450-hermes-update.md (updated 2026-10-26)
- [ 22d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md (updated 2026-10-26)
- [ 22d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
- [ 22d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md (updated 2026-10-26)
- [ 22d] runbooks/provider-health.md (updated 2026-10-26)
- [ 22d] runbooks/update-hermes-safely.md (updated 2026-10-26)
## FRESH (127)
- [ 6d] CONTRIBUTING.md
- [ 9d] agents/hermes-chaos.md
- [ 9d] agents/hermes-production.md
- [ 6d] agents/index.md
- [ 19d] agents/nanobot.md
- [ 6d] agents/open-claw.md
- [ 13d] agents/qwenpaw.md
- [ 6d] agents/zeroclaw.md
- [ 19d] comparisons/camoufox-vs-chrome.md
- [ 6d] comparisons/hermes-vs-nanobot.md
- [ 6d] comparisons/memory-backends.md
- [ 6d] comparisons/model-routers.md
- [ 19d] comparisons/steel-vs-camoufox.md
- [ 6d] concepts/agent-memory.md
- [ 19d] concepts/browser-automation.md
- [ 6d] concepts/model-routing.md
- [ 6d] concepts/persistent-shells.md
- [ 6d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 19d] decisions/index.md
- [ 6d] decisions/keep-openrouter-as-fallback.md
- [ 6d] decisions/persistent-shell-enabled.md
- [ 19d] decisions/rollback-steel-to-camofox.md
- [ 6d] experiments/index.md
- [ 5d] incidents/2026-07-29-mnemosyne-relay.md
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 5d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 12d] incidents/2026-08-15-skills-index-drift.md
- [ 13d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 6d] incidents/2026-08-29-gmail-read-state.md
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
- [ 13d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 19d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 19d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 16d] incidents/2026-10-09-memory-recall-relevance.md
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 21d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 21d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 20d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 18d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 18d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 18d] incidents/2026-10-29-conduit-voice-silence.md
- [ 17d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 18d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 17d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 16d] incidents/2026-10-31-asana-access-failures.md
- [ 16d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 16d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 15d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 14d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 13d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 11d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 11d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 12d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 8d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 12d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 10d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 8d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 0d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 6d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 6d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 5d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 4d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 0d] incidents/2026-11-15-plugin-load-timeout-storm.md
- [ 0d] index.md
- [ 10d] infrastructure/backups.md
- [ 6d] infrastructure/credentials-map.md
- [ 10d] infrastructure/docker-services.md
- [ 6d] infrastructure/domains-and-tunnels.md
- [ 10d] infrastructure/hosts.md
- [ 10d] infrastructure/network-map.md
- [ 10d] infrastructure/proxmox.md
- [ 5d] infrastructure/tailscale.md
- [ 13d] queries/retained-research-answers.md
- [ 16d] reports/uber-eats/2026-07.md
- [ 16d] reports/uber-eats/2026-08.md
- [ 16d] reports/uber-eats/2026-09.md
- [ 4d] reports/uber-eats/2026-10.md
- [ 4d] reports/uber-eats/index.md
- [ 6d] runbooks/backup-wiki.md
- [ 6d] runbooks/diagnose-docker.md
- [ 6d] runbooks/gateway-resource-resilience.md
- [ 10d] runbooks/job-radar.md
- [ 6d] runbooks/multiplexer-setup.md
- [ 19d] runbooks/recover-docker-service.md
- [ 6d] runbooks/restart-browser.md
- [ 6d] runbooks/restart-hermes.md
- [ 6d] runbooks/restore-openrouter.md
- [ 6d] runbooks/test-openrouter.md
- [ 6d] runbooks/troubleshoot-tool-loop.md
- [ 9d] systems/browser-backend.md
- [ 6d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 0d] systems/current-profile.md
- [ 6d] systems/guanaco.md
- [ 6d] systems/headless-server.md
- [ 8d] systems/hermes-agent.md
- [ 4d] systems/hermes-hq-push.md
- [ 6d] systems/hermes-ui.md
- [ 13d] systems/home-assistant.md
- [ 18d] systems/homepage-dashboard.md
- [ 13d] systems/ignis.md
- [ 0d] systems/memory-backend.md
- [ 15d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 0d] systems/model-providers.md
- [ 13d] systems/nanobot.md
- [ 6d] systems/nextcloud.md
- [ 6d] systems/obsidian.md
- [ 7d] systems/omniroute.md
- [ 6d] systems/opencode-go.md
- [ 6d] systems/openrouter.md
- [ 13d] systems/paseo.md
- [ 2d] systems/phonon-tts.md
- [ 7d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 6d] systems/terminal-backend.md
- [ 6d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 7 stale, 127 fresh, 3 no-date of 137 pages
+150
View File
@@ -0,0 +1,150 @@
# Wiki freshness audit — 2026-11-18
Threshold: >21 days since `updated`
## STALE (2)
- [ 24d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
- [ 23d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
## FRESH (132)
- [ 7d] CONTRIBUTING.md
- [ 10d] agents/hermes-chaos.md
- [ 10d] agents/hermes-production.md
- [ 7d] agents/index.md
- [ 20d] agents/nanobot.md
- [ 7d] agents/open-claw.md
- [ 14d] agents/qwenpaw.md
- [ 7d] agents/zeroclaw.md
- [ 20d] comparisons/camoufox-vs-chrome.md
- [ 7d] comparisons/hermes-vs-nanobot.md
- [ 7d] comparisons/memory-backends.md
- [ 7d] comparisons/model-routers.md
- [ 20d] comparisons/steel-vs-camoufox.md
- [ 7d] concepts/agent-memory.md
- [ 20d] concepts/browser-automation.md
- [ 7d] concepts/model-routing.md
- [ 7d] concepts/persistent-shells.md
- [ 7d] concepts/tool-calling.md
- [ 1d] current-state.md
- [ 20d] decisions/index.md
- [ 7d] decisions/keep-openrouter-as-fallback.md
- [ 7d] decisions/persistent-shell-enabled.md
- [ 20d] decisions/rollback-steel-to-camofox.md
- [ 7d] experiments/index.md
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 13d] incidents/2026-08-15-skills-index-drift.md
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 7d] incidents/2026-08-29-gmail-read-state.md
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
- [ 14d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 21d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 19d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 19d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 19d] incidents/2026-10-29-conduit-voice-silence.md
- [ 18d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 19d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 18d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 17d] incidents/2026-10-31-asana-access-failures.md
- [ 17d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 17d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 16d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 15d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 14d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 12d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 12d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 13d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 9d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 13d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 11d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 9d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 7d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 7d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 6d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 5d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 1d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 1d] incidents/2026-11-15-plugin-load-timeout-storm.md
- [ 1d] index.md
- [ 11d] infrastructure/backups.md
- [ 7d] infrastructure/credentials-map.md
- [ 11d] infrastructure/docker-services.md
- [ 7d] infrastructure/domains-and-tunnels.md
- [ 11d] infrastructure/hosts.md
- [ 11d] infrastructure/network-map.md
- [ 11d] infrastructure/proxmox.md
- [ 6d] infrastructure/tailscale.md
- [ 14d] queries/retained-research-answers.md
- [ 17d] reports/uber-eats/2026-07.md
- [ 17d] reports/uber-eats/2026-08.md
- [ 17d] reports/uber-eats/2026-09.md
- [ 5d] reports/uber-eats/2026-10.md
- [ 5d] reports/uber-eats/index.md
- [ 7d] runbooks/backup-wiki.md
- [ 7d] runbooks/diagnose-docker.md
- [ 7d] runbooks/gateway-resource-resilience.md
- [ 11d] runbooks/job-radar.md
- [ 7d] runbooks/multiplexer-setup.md
- [ 0d] runbooks/provider-health.md
- [ 20d] runbooks/recover-docker-service.md
- [ 7d] runbooks/restart-browser.md
- [ 7d] runbooks/restart-hermes.md
- [ 7d] runbooks/restore-openrouter.md
- [ 7d] runbooks/test-openrouter.md
- [ 7d] runbooks/troubleshoot-tool-loop.md
- [ 0d] runbooks/update-hermes-safely.md
- [ 10d] systems/browser-backend.md
- [ 7d] systems/cloudflare-tunnel.md
- [ 1d] systems/conduit.md
- [ 1d] systems/current-profile.md
- [ 7d] systems/guanaco.md
- [ 7d] systems/headless-server.md
- [ 9d] systems/hermes-agent.md
- [ 5d] systems/hermes-hq-push.md
- [ 7d] systems/hermes-ui.md
- [ 14d] systems/home-assistant.md
- [ 19d] systems/homepage-dashboard.md
- [ 14d] systems/ignis.md
- [ 1d] systems/memory-backend.md
- [ 16d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 1d] systems/model-providers.md
- [ 14d] systems/nanobot.md
- [ 7d] systems/nextcloud.md
- [ 7d] systems/obsidian.md
- [ 8d] systems/omniroute.md
- [ 7d] systems/opencode-go.md
- [ 7d] systems/openrouter.md
- [ 14d] systems/paseo.md
- [ 3d] systems/phonon-tts.md
- [ 8d] systems/push-notifications.md
- [ 1d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 7d] systems/terminal-backend.md
- [ 7d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 2 stale, 132 fresh, 3 no-date of 137 pages
+150
View File
@@ -0,0 +1,150 @@
# Wiki freshness audit — 2026-11-19
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (134)
- [ 8d] CONTRIBUTING.md
- [ 11d] agents/hermes-chaos.md
- [ 11d] agents/hermes-production.md
- [ 8d] agents/index.md
- [ 21d] agents/nanobot.md
- [ 8d] agents/open-claw.md
- [ 15d] agents/qwenpaw.md
- [ 8d] agents/zeroclaw.md
- [ 21d] comparisons/camoufox-vs-chrome.md
- [ 8d] comparisons/hermes-vs-nanobot.md
- [ 8d] comparisons/memory-backends.md
- [ 8d] comparisons/model-routers.md
- [ 21d] comparisons/steel-vs-camoufox.md
- [ 8d] concepts/agent-memory.md
- [ 21d] concepts/browser-automation.md
- [ 8d] concepts/model-routing.md
- [ 8d] concepts/persistent-shells.md
- [ 8d] concepts/tool-calling.md
- [ 2d] current-state.md
- [ 21d] decisions/index.md
- [ 8d] decisions/keep-openrouter-as-fallback.md
- [ 8d] decisions/persistent-shell-enabled.md
- [ 21d] decisions/rollback-steel-to-camofox.md
- [ 8d] experiments/index.md
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 14d] incidents/2026-08-15-skills-index-drift.md
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 8d] incidents/2026-08-29-gmail-read-state.md
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
- [ 13d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 7d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 4d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 20d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 20d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 20d] incidents/2026-10-29-conduit-voice-silence.md
- [ 19d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 20d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 19d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 18d] incidents/2026-10-31-asana-access-failures.md
- [ 18d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 18d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 17d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 16d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 15d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 13d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 13d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 14d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 10d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 14d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 12d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 10d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 8d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 8d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 7d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 6d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 2d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 2d] incidents/2026-11-15-plugin-load-timeout-storm.md
- [ 2d] index.md
- [ 12d] infrastructure/backups.md
- [ 8d] infrastructure/credentials-map.md
- [ 12d] infrastructure/docker-services.md
- [ 8d] infrastructure/domains-and-tunnels.md
- [ 12d] infrastructure/hosts.md
- [ 12d] infrastructure/network-map.md
- [ 12d] infrastructure/proxmox.md
- [ 7d] infrastructure/tailscale.md
- [ 15d] queries/retained-research-answers.md
- [ 0d] reports/communities-wa-social-media-audit-2026.md
- [ 18d] reports/uber-eats/2026-07.md
- [ 18d] reports/uber-eats/2026-08.md
- [ 18d] reports/uber-eats/2026-09.md
- [ 6d] reports/uber-eats/2026-10.md
- [ 6d] reports/uber-eats/index.md
- [ 8d] runbooks/backup-wiki.md
- [ 8d] runbooks/diagnose-docker.md
- [ 8d] runbooks/gateway-resource-resilience.md
- [ 12d] runbooks/job-radar.md
- [ 8d] runbooks/multiplexer-setup.md
- [ 1d] runbooks/provider-health.md
- [ 21d] runbooks/recover-docker-service.md
- [ 8d] runbooks/restart-browser.md
- [ 8d] runbooks/restart-hermes.md
- [ 8d] runbooks/restore-openrouter.md
- [ 8d] runbooks/test-openrouter.md
- [ 8d] runbooks/troubleshoot-tool-loop.md
- [ 1d] runbooks/update-hermes-safely.md
- [ 11d] systems/browser-backend.md
- [ 8d] systems/cloudflare-tunnel.md
- [ 2d] systems/conduit.md
- [ 2d] systems/current-profile.md
- [ 8d] systems/guanaco.md
- [ 8d] systems/headless-server.md
- [ 10d] systems/hermes-agent.md
- [ 6d] systems/hermes-hq-push.md
- [ 8d] systems/hermes-ui.md
- [ 15d] systems/home-assistant.md
- [ 20d] systems/homepage-dashboard.md
- [ 15d] systems/ignis.md
- [ 2d] systems/memory-backend.md
- [ 17d] systems/messaging-integrations.md
- [ 2d] systems/mnemosyne.md
- [ 2d] systems/model-providers.md
- [ 15d] systems/nanobot.md
- [ 8d] systems/nextcloud.md
- [ 8d] systems/obsidian.md
- [ 9d] systems/omniroute.md
- [ 8d] systems/opencode-go.md
- [ 8d] systems/openrouter.md
- [ 15d] systems/paseo.md
- [ 4d] systems/phonon-tts.md
- [ 9d] systems/push-notifications.md
- [ 2d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 8d] systems/terminal-backend.md
- [ 8d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 134 fresh, 3 no-date of 137 pages
+152
View File
@@ -0,0 +1,152 @@
# Wiki freshness audit — 2026-11-20
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (136)
- [ 9d] CONTRIBUTING.md
- [ 12d] agents/hermes-chaos.md
- [ 12d] agents/hermes-production.md
- [ 9d] agents/index.md
- [ 0d] agents/nanobot.md
- [ 9d] agents/open-claw.md
- [ 16d] agents/qwenpaw.md
- [ 9d] agents/zeroclaw.md
- [ 0d] comparisons/camoufox-vs-chrome.md
- [ 9d] comparisons/hermes-vs-nanobot.md
- [ 9d] comparisons/memory-backends.md
- [ 9d] comparisons/model-routers.md
- [ 0d] comparisons/steel-vs-camoufox.md
- [ 9d] concepts/agent-memory.md
- [ 0d] concepts/browser-automation.md
- [ 9d] concepts/model-routing.md
- [ 9d] concepts/persistent-shells.md
- [ 9d] concepts/tool-calling.md
- [ 0d] current-state.md
- [ 0d] decisions/index.md
- [ 9d] decisions/keep-openrouter-as-fallback.md
- [ 9d] decisions/persistent-shell-enabled.md
- [ 0d] decisions/rollback-steel-to-camofox.md
- [ 9d] experiments/index.md
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 13d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 15d] incidents/2026-08-15-skills-index-drift.md
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 9d] incidents/2026-08-29-gmail-read-state.md
- [ 8d] incidents/2026-09-02-sftpgo-outage.md
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 2d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 10d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 2d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 9d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 1d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 21d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 21d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 21d] incidents/2026-10-29-conduit-voice-silence.md
- [ 20d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 21d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 20d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 19d] incidents/2026-10-31-asana-access-failures.md
- [ 19d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 19d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 18d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 17d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 16d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 14d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 14d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 15d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 11d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 15d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 13d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 11d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 3d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
- [ 9d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
- [ 9d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
- [ 8d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
- [ 7d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
- [ 3d] incidents/2026-11-15-plugin-load-timeout-storm.md
- [ 0d] incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
- [ 0d] incidents/2026-11-18-wiki-audit-credit-outage.md
- [ 0d] index.md
- [ 13d] infrastructure/backups.md
- [ 9d] infrastructure/credentials-map.md
- [ 13d] infrastructure/docker-services.md
- [ 9d] infrastructure/domains-and-tunnels.md
- [ 13d] infrastructure/hosts.md
- [ 13d] infrastructure/network-map.md
- [ 13d] infrastructure/proxmox.md
- [ 8d] infrastructure/tailscale.md
- [ 16d] queries/retained-research-answers.md
- [ 1d] reports/communities-wa-social-media-audit-2026.md
- [ 19d] reports/uber-eats/2026-07.md
- [ 19d] reports/uber-eats/2026-08.md
- [ 19d] reports/uber-eats/2026-09.md
- [ 7d] reports/uber-eats/2026-10.md
- [ 7d] reports/uber-eats/index.md
- [ 9d] runbooks/backup-wiki.md
- [ 9d] runbooks/diagnose-docker.md
- [ 9d] runbooks/gateway-resource-resilience.md
- [ 13d] runbooks/job-radar.md
- [ 9d] runbooks/multiplexer-setup.md
- [ 2d] runbooks/provider-health.md
- [ 0d] runbooks/recover-docker-service.md
- [ 9d] runbooks/restart-browser.md
- [ 9d] runbooks/restart-hermes.md
- [ 9d] runbooks/restore-openrouter.md
- [ 9d] runbooks/test-openrouter.md
- [ 9d] runbooks/troubleshoot-tool-loop.md
- [ 2d] runbooks/update-hermes-safely.md
- [ 12d] systems/browser-backend.md
- [ 9d] systems/cloudflare-tunnel.md
- [ 3d] systems/conduit.md
- [ 3d] systems/current-profile.md
- [ 9d] systems/guanaco.md
- [ 9d] systems/headless-server.md
- [ 11d] systems/hermes-agent.md
- [ 7d] systems/hermes-hq-push.md
- [ 9d] systems/hermes-ui.md
- [ 16d] systems/home-assistant.md
- [ 21d] systems/homepage-dashboard.md
- [ 16d] systems/ignis.md
- [ 3d] systems/memory-backend.md
- [ 18d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 3d] systems/model-providers.md
- [ 16d] systems/nanobot.md
- [ 9d] systems/nextcloud.md
- [ 9d] systems/obsidian.md
- [ 10d] systems/omniroute.md
- [ 9d] systems/opencode-go.md
- [ 9d] systems/openrouter.md
- [ 16d] systems/paseo.md
- [ 5d] systems/phonon-tts.md
- [ 10d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 9d] systems/terminal-backend.md
- [ 9d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 136 fresh, 3 no-date of 139 pages
@@ -3,8 +3,8 @@ title: Department of Communities WA Social Media Audit and First-Day Brief
type: report
status: active
created: 2026-08-28
updated: 2026-10-25
verified_on: 2026-10-25
updated: 2026-11-19
verified_on: 2026-11-19
review_after: 2026-11-26
confidence: medium
tags: [content, research, department-of-communities, social-media]
@@ -37,6 +37,13 @@ point-in-time public-source figures from the 2026-08-28 audit and remain as reco
not re-counted. The `review_after` date of 2026-11-26 stands for a full refresh against native
analytics.
**Re-verified 2026-11-19 (stocktake).** Live check of wa.gov.au/organisation/department-of-communities
(page last updated 18 November 2026): the remit text is **verbatim-identical** to the quote above,
all **five Ministers** are still listed (McGurk, Winton, Beazley, Stojkovski, Swinbourn), and the
announcement flow (child-safety package 18 Nov, Seniors Card guide 12 Nov, Seniors Week 11 Nov)
confirms the recorded content pillars remain active topics. Structural claims of this page all
still hold.
## Strategic baseline
- The Department's current remit spans child protection, community services, homelessness, prevention of family and domestic violence, disability services, early childhood education and care regulation, women's interests, youth, seniors and ageing, carers, and volunteering.
+4 -2
View File
@@ -3,8 +3,8 @@ title: Provider Health Check
type: runbook
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
last_tested: 2026-10-26
confidence: high
tags: [runbook, providers, health, routing]
@@ -58,6 +58,8 @@ with urllib.request.urlopen(req, timeout=5) as r:
```
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
> **Re-executed 2026-11-18:** step-3 snippet OK — CT600 `/models` → 200, **6,501 models** (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 `/api/health` → 200; `omni-101` `/api/health` → 200; fallback endpoints `opencode.ai` / `inference-api.nousresearch.com` / `ollama.com` all → 200.
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
+7 -2
View File
@@ -3,8 +3,8 @@ title: Recover a Docker service
type: runbook
status: active
created: 2026-07-22
updated: 2026-10-29
verified_on: 2026-10-29
updated: 2026-11-20
verified_on: 2026-11-20
tags: [runbook, docker, recovery]
sources: []
confidence: high
@@ -20,6 +20,11 @@ Container shows healthy after recovery probe reboots.
## Last successfully used
2026-07-22
> **Re-verified 2026-11-20 (stocktake, procedure only).** `docker ps` as `hermes` on CT460 still
> returns permission denied (`/var/run/docker.sock` is `root:docker 0660`) — sudo or SSH to the
> owning host remains required. CT201 homepage degradation **persists**: `:8085` answers **502**
> locally, `homepage-manager` up but no dashboard container on 8082. Steel remains fully removed
> (re-probed CT201/CT333: no containers, no images, no routes).
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
+4 -2
View File
@@ -3,8 +3,8 @@ title: Update Hermes safely
type: runbook
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-18
verified_on: 2026-11-18
last_tested: 2026-10-26
confidence: high
tags: [runbook, hermes, update]
@@ -28,6 +28,8 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
- Available disk space in Hermes install and backup paths
- No active Hermes chat/webui shell issues before starting
> **Re-verified 2026-11-18:** prerequisites hold on CT460 — `hermes --version` → `v0.23.0` from the git install at `/home/hermes/.hermes/hermes-agent`, legacy `venv/` present, three managed runtime venvs present under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. (CT450 currently runs `v0.23.1` — see [[incidents/2026-10-03-ct450-hermes-update]].)
## Procedure
### 1. Create a backup first
+15 -2
View File
@@ -3,8 +3,8 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-11-09
verified_on: 2026-11-09
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-12-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
@@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
### 2026-11-15 — 474 registrations dropped across eight failed loads
`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after
10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called
register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events
(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations,
so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the
caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out
**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts:
[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side:
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
## Open items
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
+16 -16
View File
@@ -3,9 +3,9 @@ title: Current Hermes Profile
type: system
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
review_after: 2026-11-09
updated: 2026-11-17
verified_on: 2026-11-17
review_after: 2026-12-01
confidence: high
tags: [system, hermes, profile]
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
@@ -16,9 +16,9 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
## Active Profile
- **Name:** `default`
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
- **Other profiles present:** none — `~/.hermes/profiles/` holds only a `.deleted/` marker. The `ops` (Tusk elephant) profile was **deleted 2026-10-11**; two chats still routed to it afterwards, so expect stray references. Other profiles do **not** affect this session; see the profile-isolation note below.
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
## Key Settings (re-verified against live config 2026-11-17)
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
@@ -38,24 +38,24 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
credential** — a fallover would try it first.
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
- **⚠ Cron pins (re-verified live from `jobs.json` 2026-11-17).** **21 of the 26 enabled jobs pin OmniRoute
explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1); the other 5 enabled jobs carry no
pin (no-agent/script jobs) and follow the global default, which is OmniRoute — the fleet runs a **single
upstream path**. → [[systems/scheduled-tasks]]
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
`model.fallbacks` does not exist.
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
[[systems/model-providers]].
- ⚠ The `custom:omni` provider key is **no longer configured** and **no jobs name it any more** (verified from
`jobs.json` 2026-11-17; two jobs still did on 2026-10-26) — see [[systems/model-providers]].
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
### Other settings (re-verified live 2026-11-17)
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13. A ninth slot, **`pulse_memory`**, now points at `nous` / `anthropic/claude-haiku-4.5` — i.e. the **dead nous credential**. → [[systems/model-providers]]
- **Switch persistence:** `persist_switch_by_default: false`
- **Timezone:** `Australia/Perth`
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
- **Config version:** `_config_version: 46`; config file ≈514 KB / ≈13,200 lines
- **MCP servers:** 5 configured
- **Plugins enabled:** 11
- **Config version:** `_config_version: 52` (config file ≈514 KB / ≈13,200 lines as of 2026-10-26)
- **MCP servers:** 6 configured (`Not human search`, `asana` (disabled), `cua-driver`, `hound`, `mcp-portal`, `approved_shared_memory`)
- **Plugins enabled:** 15
- **Pet:** configured slug `rhino`, but `enabled: false` (render mode `unicode`, scale 0.55)
- **Skin:** `slate`
- **Personality:** empty string in config — the active persona comes from `SOUL.md`, not this key
+4 -4
View File
@@ -3,8 +3,8 @@ title: Memory Backend
type: system
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-17
verified_on: 2026-11-17
confidence: high
tags: [memory, mnemosyne, agent]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -34,7 +34,7 @@ Persist and recall agent memory across sessions.
- **Sleep threshold:** 50 turns
- **Nudge interval:** 10
- **Ignore patterns:** none set
- **Live totals (2026-10-26):** 2,350 memories — 842 working / 1,090 episodic / 418 `memoria_facts`
- **Live totals (2026-11-17, sqlite):** 2,901 memories — 1,130 working / 1,249 episodic / 522 `memoria_facts`
## Storage Location
- Mnemosyne home: `~/.hermes/mnemosyne/`
@@ -48,7 +48,7 @@ Persist and recall agent memory across sessions.
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Embedding coverage re-verified 2026-10-26** — `check_embedding_coverage.py` reports 1,899 / 2,350 (80.8 %) embedded; **working 815 / 842 (96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)**. Both tiers are **above the 95 % reindex threshold**, so no reindex was triggered. `fastembed` 0.8.1 is present in **both** managed runtimes and 0.8.0 in the legacy venv; `FASTEMBED_CACHE_PATH` is pinned. The 2026-10-03 follow-up (assert `fastembed` in *all* managed runtimes) is now satisfied.
- **Embedding coverage re-verified 2026-11-17** — `check_embedding_coverage.py` reports 2,379 / 2,901 (82.0 %) embedded; **working 1,130 / 1,130 (100 %)** and **episodic 1,249 / 1,249 (100 %)** (the gap is `memoria_facts`, unvectorized by design). Both tiers are **above the 95 % reindex threshold**, so no reindex is needed. Earlier (2026-10-26) check: 1,899 / 2,350 (80.8 %), working 96.8 %, episodic 99.4 %.
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
+9 -4
View File
File diff suppressed because one or more lines are too long
+8 -6
View File
@@ -3,8 +3,8 @@ title: Model Providers
type: system
status: active
created: 2026-07-22
updated: 2026-10-26
verified_on: 2026-10-26
updated: 2026-11-17
verified_on: 2026-11-17
confidence: high
tags: [model, provider, inference, routing]
sources: [live-probe, config, web-search]
@@ -20,7 +20,7 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
| Provider | URL | Models | Role |
|---|---|---|---|
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 21 of 26 enabled jobs pin it (2026-11-17) |
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
@@ -53,13 +53,15 @@ Provider names that appear in job configs but not in `custom_providers` do **not
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
- **21 of 26 enabled scheduled jobs pin OmniRoute explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1; verified from live `jobs.json` 2026-11-17). The remaining 5 enabled jobs carry no pin (no-agent/script jobs) and follow the global default, which is **also** OmniRoute — the fleet runs a **single upstream path**. The two jobs that still named the removed `custom:omni` key on 2026-10-26 no longer do. → [[systems/scheduled-tasks]]
## Auxiliary Role Routing (re-verified live 2026-10-26)
## Auxiliary Role Routing (re-verified live 2026-11-17)
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
**Compression timeout 90 s; vision timeout 180 s.**
**Compression timeout 90 s; vision timeout 180 s.** A ninth slot, **`pulse_memory`**, is
**`nous` / `anthropic/claude-haiku-4.5`** — i.e. it sits on the **dead `nous` credential**; a pulse-memory call
that hits it will fail until the slot or the credential is fixed.
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
+14 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-15
verified_on: 2026-11-15
review_after: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
review_after: 2026-11-21
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,17 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks
## Count
- **Current as of 2026-11-20 00:5x (three-window audit — 2026-11-17, 11-18 and 11-19; the audits slotted at 11-18 00:00 and 11-19 00:00 both died, so 11-17 and 11-18 had never been audited; live `jobs.json` + `executions.db` + `deliveries.db` + `cron_incidents` + `mnemosyne.db`).** Ledger: **11-17 — 173 rows, 171 `completed` / 2 `failed`** (both `ea79a2107e36` *Mnemosyne Health Monitor*, `00:01` and `12:01`); **11-18 — 172 rows, 153 / 19** (this audit `00:00:08 → 00:01:43`, the monitor twice, and **16** *Mnemosyne Watchdog* slots from `20:01:06`); **11-19 — 176 rows, 78 / 98** — **the worst day in the retained window** (audit `00:00:16 → 00:03:02`, monitor `12:01`, **96** watchdog slots); **11-20 so far — 8 rows, 3 `completed` / 4 `failed` / 1 `running`** (this audit). Deliveries since 11-18: **24 — 23 `delivered` (9 `for_failure: 1`) / 1 `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window at the same count** (following the 11th, recorded on 11-17; the 11-18 and 11-19 windows went unrecorded because both audits died) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` Wiki Audit `error`, `failure_streak` **2** (last clean run 11-17), `96eb95ce12f2` Watchdog `error`, streak **115**, `ea79a2107e36` Health Monitor `error`, streak **2**, `08eb86f02d37` Gmail Action Digest `delivery_failed`, streak **0**. **Two of the three jobs that keep the fleet safe are themselves red — and the 03:00 stocktake is not:** `1a8e90ea2271` ran `completed` on 11-17 (**11 m 35 s**), 11-18 (**23 m 07 s**) and 11-19 (**24 m 38 s**), so the estate's structural maintenance kept running while both alarm paths failed. `cron_incidents` unclosed **5 → 4**, none closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, **`alerted_at: None`** — the audit's silent skip), `96eb95_98ade422d195` (*script*, 2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16), `ea79a2_f2be2db3bfce` (2026-11-19 12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36) — both monitor rows on the same `STALE MEMORY PIPELINE` metric that is now measuring a *quiet estate* rather than a broken capture (see [[systems/mnemosyne]]). Cadence and pins unchanged; this audit still runs `custom:omni-101` / `deepseek-paid-smart`, and the Mnemosyne Watchdog's model/provider fields remain unset (inherits global). Full detail: [[incidents/2026-11-18-wiki-audit-credit-outage]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]].
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
failures and **0** hook timeouts, against 52 and 13 the day before — see
[[incidents/2026-11-15-plugin-load-timeout-storm]].
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-15
verified_on: 2026-11-15
updated: 2026-11-20
verified_on: 2026-11-20
confidence: high
tags: [skills, hermes]
sources: []