Compare commits
9
Commits
402d8546e8
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f603c29d17 | ||
|
|
8bf0a37022 | ||
|
|
226767f3ec | ||
|
|
16f2c878c0 | ||
|
|
98610466cc | ||
|
|
4574c4ebc3 | ||
|
|
e5612c8a8d | ||
|
|
4eee02b9a2 | ||
|
|
9f80926dda |
+40
-15
@@ -3,8 +3,8 @@ title: Nanobot
|
||||
type: agent
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [agent, nanobot, homelab]
|
||||
sources: [live-probe, runtime-config]
|
||||
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
||||
- **Workspace:** `/home/nano/.nanobot/workspace`
|
||||
|
||||
## Current state
|
||||
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening.
|
||||
- **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
|
||||
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
|
||||
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
|
||||
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
|
||||
@@ -38,22 +38,31 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
||||
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
|
||||
|
||||
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
|
||||
`gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`.
|
||||
~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
|
||||
|
||||
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
## Cron jobs
|
||||
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded:
|
||||
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST
|
||||
- `morning-quota-report` — daily at 07:30 AWST
|
||||
- `daily-quota-730am` — daily at 07:30 AWST
|
||||
- `vault-daily-seed` — daily at 08:00 AWST
|
||||
- `daily-agent-briefing` — daily at 08:15 AWST
|
||||
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST
|
||||
- `daily-quota-7pm` — daily at 19:00 AWST
|
||||
- `evening-quota-report` — daily at 21:00 AWST
|
||||
- `heartbeat` — every 30m
|
||||
- `dream` — every 2h (memory consolidation)
|
||||
**Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
|
||||
(all schedules Australia/Perth):
|
||||
|
||||
| Job | Schedule | Enabled |
|
||||
|---|---|---|
|
||||
| `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
|
||||
| `nightly-skills-export-sync` | daily 04:00 | ✅ |
|
||||
| `morning-news-podcast` | daily 07:00 | ✅ |
|
||||
| `daily-quota-730am` | daily 07:30 | ✅ |
|
||||
| `morning-quota-report` | daily 07:30 | ❌ disabled |
|
||||
| `vault-daily-seed` | daily 08:00 | ✅ |
|
||||
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
|
||||
| `daily-quota-7pm` | daily 19:00 | ✅ |
|
||||
| `evening-quota-report` | daily 21:00 | ❌ disabled |
|
||||
| `heartbeat` | every 30m | ✅ |
|
||||
| `dream` | every 2h (memory consolidation) | ✅ |
|
||||
|
||||
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
|
||||
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
|
||||
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
|
||||
|
||||
## Skills
|
||||
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
|
||||
@@ -73,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
|
||||
- [[runbooks/restart-hermes]]
|
||||
- [[runbooks/nanobot-restart]]
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
|
||||
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
|
||||
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
|
||||
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
|
||||
> agentmail-daemon and readlater (:3210) processes all still run.
|
||||
>
|
||||
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
|
||||
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
|
||||
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
|
||||
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
|
||||
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
|
||||
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
|
||||
>
|
||||
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
|
||||
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
|
||||
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
|
||||
## Related
|
||||
- [[comparisons/hermes-vs-nanobot]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: medium
|
||||
tags: [comparison, browser]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -46,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
|
||||
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
||||
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
|
||||
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
|
||||
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
|
||||
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
|
||||
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
|
||||
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
|
||||
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
|
||||
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
|
||||
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
|
||||
> No Chrome/Chromium process found in this probe either.
|
||||
### Chrome (not found in use)
|
||||
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
|
||||
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
|
||||
type: comparison
|
||||
status: active
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [comparison, browser, steel, camofox, decision]
|
||||
sources: [live-probe, session-history]
|
||||
@@ -77,6 +77,13 @@ the CT201 and CT333 deployments torn down — so returning to Steel means redepl
|
||||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
||||
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
|
||||
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
|
||||
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
|
||||
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
|
||||
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
|
||||
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
|
||||
> `cloud_provider: camofox`. All removal claims **hold**.
|
||||
## Related
|
||||
- [[systems/browser-backend]]
|
||||
- [[concepts/browser-automation]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Browser Automation
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [concept, browser, camofox, steel]
|
||||
sources: [live-probe, session-history]
|
||||
@@ -33,6 +33,10 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
|
||||
|
||||
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
|
||||
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
|
||||
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
|
||||
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
|
||||
## Related
|
||||
- [[systems/browser-backend]]
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
|
||||
+77
-4
@@ -3,16 +3,89 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-11-15
|
||||
Updated: 2026-11-20
|
||||
|
||||
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
|
||||
|
||||
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
|
||||
|
||||
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
|
||||
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
|
||||
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
|
||||
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
|
||||
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
|
||||
|
||||
**Previous window:**
|
||||
|
||||
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
|
||||
|
||||
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
|
||||
load-timeout storm nobody had logged.**
|
||||
|
||||
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
|
||||
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
|
||||
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
|
||||
streak **120** cycles. No cause, no fix, `status: active` →
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
|
||||
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
|
||||
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
|
||||
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
|
||||
cleared with no fix applied. The companion
|
||||
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
|
||||
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
|
||||
a *larger* `state.db`).
|
||||
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
|
||||
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
|
||||
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
|
||||
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
|
||||
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
|
||||
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
|
||||
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
|
||||
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
|
||||
|
||||
|
||||
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
||||
|
||||
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
|
||||
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
|
||||
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
|
||||
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
|
||||
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
|
||||
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
|
||||
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
|
||||
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
|
||||
all three alerts read alike. **Do not read it as the stall length.**
|
||||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
|
||||
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
|
||||
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
|
||||
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
|
||||
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
|
||||
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
|
||||
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
|
||||
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
|
||||
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
|
||||
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
|
||||
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
|
||||
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
|
||||
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
|
||||
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
|
||||
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
|
||||
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
|
||||
human-side for the wiki to have missed.
|
||||
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
|
||||
no-date of 136** pages.
|
||||
|
||||
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
||||
|
||||
|
||||
+6
-2
@@ -3,8 +3,8 @@ title: Decisions
|
||||
type: collection
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [decisions]
|
||||
---
|
||||
@@ -31,6 +31,10 @@ Stable strategic and config decisions, with evidence and status.
|
||||
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
|
||||
> backend.
|
||||
|
||||
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
|
||||
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
|
||||
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
|
||||
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
|
||||
## History
|
||||
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
||||
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
|
||||
type: decision
|
||||
status: active
|
||||
created: 2026-08-02
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [decision, browser, steel, camofox, rollback]
|
||||
sources: [session-history, live-config]
|
||||
@@ -50,7 +50,7 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
||||
> restoration was needed or wanted.
|
||||
|
||||
## Rationale
|
||||
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||
1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
||||
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
|
||||
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
|
||||
@@ -103,6 +103,14 @@ removed**, so the `plugin_steel` toolset is no longer selectable.
|
||||
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
|
||||
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
|
||||
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
|
||||
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
|
||||
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
|
||||
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
|
||||
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
|
||||
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
|
||||
> The decision and its consequences are unchanged.
|
||||
## Related
|
||||
- [[comparisons/steel-vs-camoufox]]
|
||||
- [[systems/browser-backend]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||
@@ -130,6 +130,12 @@ done.
|
||||
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
||||
name is absent from the provider's live catalog.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
|
||||
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
|
||||
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
|
||||
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
|
||||
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
|
||||
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
confidence: high
|
||||
tags: [hermes, ct450, chaos, update, incident]
|
||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||
@@ -49,7 +49,10 @@ Two gotchas that cost time:
|
||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||
|
||||
## Current state (re-verified 2026-10-26)
|
||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
|
||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
|
||||
|
||||
## Current state (re-verified 2026-11-18)
|
||||
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
|
||||
|
||||
## Follow-ups
|
||||
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-04
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||
@@ -65,6 +65,9 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
|
||||
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||
figures below are superseded.
|
||||
|
||||
## Current state (re-verified 2026-11-18)
|
||||
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
|
||||
|
||||
## Follow-ups
|
||||
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
|
||||
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-06
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, model-pin, omniroute, incident, resolved]
|
||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
|
||||
@@ -119,6 +119,17 @@ state rather than left on trust.
|
||||
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
|
||||
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
|
||||
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
|
||||
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
|
||||
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
|
||||
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
|
||||
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
|
||||
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
|
||||
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
|
||||
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
|
||||
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
|
||||
> [[systems/scheduled-tasks]], not here.
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
@@ -35,6 +35,8 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
|
||||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||||
> shut, the single-node exposure is not.
|
||||
|
||||
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
|
||||
|
||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-21
|
||||
updated: 2026-10-27
|
||||
verified_on: 2026-10-27
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-03
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
|
||||
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
|
||||
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
|
||||
| ~09:10 | the orphaned execution row closes as **`unknown`** |
|
||||
|
||||
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
|
||||
|
||||
## Impact
|
||||
|
||||
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Conduit Call Requests Refused by the Agent
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-25
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-19
|
||||
verified_on: 2026-11-19
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
|
||||
@@ -93,3 +93,11 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
||||
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||
- **Status stays open**: the request path is untested, not confirmed good.
|
||||
|
||||
## Status update — 2026-11-19 (stocktake re-probe)
|
||||
|
||||
- `~/.hermes/conduit-calls.json` → `history` is **still empty (0 entries)** as of 2026-11-19 03:0x
|
||||
AWST — the call path has now gone unused for ~26 days since the fault. The "verify a real call
|
||||
end-to-end" open item remains outstanding.
|
||||
- No new refusal episodes have been recorded against this page. **Status stays open** on the same
|
||||
basis as before: untested, not confirmed good.
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||
type: incident
|
||||
status: open
|
||||
status: resolved
|
||||
created: 2026-10-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
review_after: 2026-11-09
|
||||
confidence: medium
|
||||
tags: [cron, automation, hermes, incident, unresolved]
|
||||
tags: [cron, automation, hermes, incident]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||
---
|
||||
|
||||
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
|
||||
execution row.
|
||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||
|
||||
## Resolution (verified 2026-11-18)
|
||||
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||
type: incident
|
||||
status: active
|
||||
status: resolved
|
||||
created: 2026-10-27
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
review_after: 2026-11-03
|
||||
updated: 2026-11-19
|
||||
verified_on: 2026-11-19
|
||||
review_after: 2026-12-19
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
|
||||
@@ -111,3 +111,15 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
|
||||
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
||||
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
||||
(`timed_out=True` at 1.35 s of a 30 s budget).
|
||||
|
||||
## Update — 2026-11-19 (stocktake re-probe)
|
||||
|
||||
- **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3`
|
||||
`resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13**
|
||||
(live probe of `cron_incidents` in `~/.hermes/cron/executions.db`).
|
||||
- The `lost its durable fire claim ownership` error string has **no rows in the retained
|
||||
`executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention
|
||||
floor; the 10-28 update already confirmed the 10-27 window clean).
|
||||
- **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups
|
||||
1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics)
|
||||
remain open design questions and were **not** actioned.
|
||||
|
||||
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-09
|
||||
updated: 2026-11-14
|
||||
verified_on: 2026-11-14
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-20
|
||||
confidence: high
|
||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
|
||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||
|
||||
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||
|
||||
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||
|
||||
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||
rather than a mechanism.
|
||||
|
||||
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/conduit]]
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
---
|
||||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
||||
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-15
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||
---
|
||||
|
||||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
||||
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||
|
||||
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
|
||||
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
|
||||
> not a broken hook. Read the newest re-read section before the earlier readings.
|
||||
|
||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||
@@ -21,6 +25,181 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
|
||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||
the same condition was classified informational — see below).
|
||||
|
||||
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||||
|
||||
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||||
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||||
|
||||
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||||
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||||
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||||
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||||
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||||
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||||
then both stopped inside the same minute.
|
||||
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||||
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||||
observation, no `last_seen_at` accumulation, no escalation:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||||
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||||
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||||
|
||||
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||||
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||||
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||||
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||||
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||||
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||||
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||||
**re-opens the existing row** instead of fragmenting.
|
||||
|
||||
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||||
|
||||
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||||
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||||
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||||
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||||
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||||
*conversational* message — not the age of the newest capture.**
|
||||
|
||||
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||||
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||||
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||||
|
||||
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||||
| --- | --- | --- | --- |
|
||||
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||||
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||||
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||||
|
||||
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||||
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||||
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||||
that is not one.
|
||||
|
||||
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||||
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||||
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||||
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||||
|
||||
### The consolidator is alive and every cycle is a deliberate no-op
|
||||
|
||||
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||||
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||||
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||||
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||||
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||||
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||||
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||||
what would have caught this at 10:04 on 11-14.**
|
||||
|
||||
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||||
|
||||
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||||
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||||
|
||||
```
|
||||
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||||
```
|
||||
|
||||
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||||
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||||
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||||
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||||
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||||
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||||
times out at 600 s belongs next to this stall rather than dismissed.
|
||||
|
||||
|
||||
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||||
|
||||
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||||
total.**
|
||||
|
||||
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||||
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||||
**62 h 54 m** with **zero** writes.
|
||||
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||||
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||||
|
||||
| Incident id | `first_seen_at` | `alerted_at` |
|
||||
| --- | --- | --- |
|
||||
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||||
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||||
|
||||
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||||
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||||
on any single row.
|
||||
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||||
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||||
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||||
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||||
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||||
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||||
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||||
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||||
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||||
reason the stall cannot self-heal.
|
||||
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||||
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||||
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||
|
||||
|
||||
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
|
||||
|
||||
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
|
||||
|
||||
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
|
||||
| --- | --- | --- |
|
||||
| `working_memory` | 1130 | **1131** |
|
||||
| `episodic_memory` | 1249 | **1250** |
|
||||
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
|
||||
|
||||
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
|
||||
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
|
||||
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
|
||||
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
|
||||
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
|
||||
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
|
||||
|
||||
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
|
||||
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
|
||||
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
|
||||
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
|
||||
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
|
||||
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
|
||||
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
|
||||
a broken capture path — is the durable defect on this page.
|
||||
|
||||
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
|
||||
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
|
||||
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
|
||||
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
|
||||
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
|
||||
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
|
||||
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
|
||||
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
|
||||
|
||||
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
|
||||
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
|
||||
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
|
||||
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
|
||||
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
|
||||
applied**.
|
||||
|
||||
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
|
||||
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
|
||||
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
|
||||
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
|
||||
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
|
||||
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
|
||||
|
||||
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||
|
||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||
|
||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||
@@ -103,6 +282,11 @@ the loudest signal available and the only one nothing watches.**
|
||||
|
||||
## Impact
|
||||
|
||||
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||||
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||||
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||||
|
||||
|
||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||
window. Recall over that period will answer from nothing.
|
||||
@@ -133,7 +317,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
|
||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
||||
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||||
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||||
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||||
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||||
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||||
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
type: incident
|
||||
status: unresolved
|
||||
created: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-24
|
||||
confidence: high
|
||||
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||
---
|
||||
|
||||
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||
|
||||
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||
`errors.log` grep, not an estimate.
|
||||
|
||||
## What the log records (2026-11-15)
|
||||
|
||||
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||
|
||||
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||
older lines):
|
||||
|
||||
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
### By plugin (2026-11-15)
|
||||
|
||||
| Plugin | Failed loads | Registrations dropped |
|
||||
| --- | --- | --- |
|
||||
| `hermes-hq-push` | 26 | 0 |
|
||||
| `conduit_push` | 9 | **474** |
|
||||
| `homeassistant` | 7 | 34 |
|
||||
| `browser-browser-use` | 3 | 3 |
|
||||
| `basic` | 2 | 2 |
|
||||
| `cadu-secrets-vault` | 1 | 4 |
|
||||
| `telegram-platform` | 1 | 1 |
|
||||
| `nous` | 1 | 0 |
|
||||
| `drain` | 1 | 0 |
|
||||
| `browser-browserbase` | 1 | 1 |
|
||||
| **Total** | **52** | **519** |
|
||||
|
||||
## The dropped registrations are the more serious half
|
||||
|
||||
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||
[[systems/conduit]].
|
||||
|
||||
## Clustering
|
||||
|
||||
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||
spread over hours 01, 04–11 and 13:
|
||||
|
||||
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||
are refused in the same session (`local-989b3642…`).
|
||||
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||
dropping ~52–56 registrations.
|
||||
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||
failure alongside it.
|
||||
|
||||
## What this is — and is not
|
||||
|
||||
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||
**saturation symptom** on this host, not a wrong setting.
|
||||
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||
guard also blew its **600 s** budget that day (see
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||
exceeded in the same 24 h.
|
||||
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||
predict it**.
|
||||
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||
|
||||
## Open question
|
||||
|
||||
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||
it is Anthony's call).
|
||||
|
||||
## See also
|
||||
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||
---
|
||||
|
||||
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||
|
||||
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||
|
||||
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||
re-reported every 15 minutes, not a series of separate faults.
|
||||
|
||||
## The failure is a fixed, self-describing threshold trip
|
||||
|
||||
```
|
||||
Script exited with code 1
|
||||
stdout:
|
||||
🚨 Many unresolved conflicts: 59 (threshold: 50)
|
||||
```
|
||||
|
||||
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
|
||||
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
|
||||
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
|
||||
|
||||
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
|
||||
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
|
||||
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
|
||||
re-alerts forever.
|
||||
|
||||
## Alerting works — and that is the problem
|
||||
|
||||
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
|
||||
|
||||
| delivered | execution |
|
||||
|---|---|
|
||||
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
|
||||
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
|
||||
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
|
||||
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
|
||||
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
|
||||
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
|
||||
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
|
||||
|
||||
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
|
||||
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
|
||||
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
|
||||
arriving on the same job is indistinguishable from the standing one.
|
||||
|
||||
## What the 59 conflicts actually are
|
||||
|
||||
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
|
||||
unresolved.** Unresolved by creation day:
|
||||
|
||||
| created | unresolved |
|
||||
|---|---|
|
||||
| 2026-10-31 | 11 |
|
||||
| 2026-11-02 | 2 |
|
||||
| 2026-11-03 | 12 |
|
||||
| 2026-11-06 | 1 |
|
||||
| 2026-11-07 | 2 |
|
||||
| 2026-11-08 | 1 |
|
||||
| 2026-11-12 | 13 |
|
||||
| 2026-11-13 | 3 |
|
||||
| **2026-11-18** | **14** |
|
||||
| **total** | **59** |
|
||||
|
||||
Two things stand out:
|
||||
|
||||
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
|
||||
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
|
||||
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
|
||||
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
|
||||
applied here**.
|
||||
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
|
||||
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
|
||||
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
|
||||
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
|
||||
not opened in this window.
|
||||
|
||||
## The 2026-11-18 batch coincides with the migration session
|
||||
|
||||
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
|
||||
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
|
||||
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
|
||||
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
|
||||
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
|
||||
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||
the fact pairs were not inspected.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||
- The watchdog's cadence, signature construction and alert path were left as-is.
|
||||
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
|
||||
Same family as the zero-item-consolidation streak problem on
|
||||
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
|
||||
the gate tests a level, not a change.
|
||||
|
||||
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-20
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||
---
|
||||
|
||||
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
|
||||
|
||||
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
|
||||
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
|
||||
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
|
||||
reads `running` (started `2026-11-20T00:00:08`).
|
||||
|
||||
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
|
||||
|
||||
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
|
||||
|
||||
| slot | started | finished / last seen | duration | status |
|
||||
|---|---|---|---|---|
|
||||
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
|
||||
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
|
||||
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
|
||||
|
||||
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
|
||||
|
||||
```
|
||||
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
|
||||
make this request. Please purchase more credits to continue using the service. (HTTP 400);
|
||||
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
|
||||
openai-compati…
|
||||
```
|
||||
|
||||
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
|
||||
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
|
||||
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
|
||||
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
|
||||
|
||||
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
|
||||
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
|
||||
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
|
||||
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
|
||||
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
|
||||
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
|
||||
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
|
||||
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
|
||||
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
|
||||
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
|
||||
So the outage produced **no push to Anthony** and the red cron row was the only signal.
|
||||
|
||||
## Why this matters beyond one job
|
||||
|
||||
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
|
||||
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
|
||||
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
|
||||
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
|
||||
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
|
||||
|
||||
## Precedent — and a repeating pattern
|
||||
|
||||
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
|
||||
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
|
||||
|
||||
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
|
||||
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
|
||||
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
|
||||
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||
rather than at the alert throttle.
|
||||
|
||||
## Not applied / open
|
||||
|
||||
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||
cron entry was edited.
|
||||
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
|
||||
entry appended to `log.md` the same night).
|
||||
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
|
||||
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
|
||||
basis: the truncated 314-char error string.
|
||||
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
|
||||
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
|
||||
|
||||
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
|
||||
@@ -1782,3 +1782,111 @@ provider, so the cold-start path is untested. Page → `status: degraded` with a
|
||||
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
|
||||
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
||||
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
|
||||
|
||||
|
||||
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
|
||||
|
||||
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
|
||||
|
||||
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
|
||||
|
||||
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
|
||||
|
||||
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
|
||||
|
||||
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
|
||||
|
||||
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
|
||||
|
||||
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
|
||||
|
||||
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
|
||||
## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST)
|
||||
|
||||
**Verdict: 1 new page + 7 page updates, nothing archived, no new system.**
|
||||
|
||||
**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173
|
||||
`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`);
|
||||
deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15
|
||||
watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load
|
||||
6.19–7.48. **No new system, no new concept, no page worth archiving.**
|
||||
|
||||
**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous
|
||||
audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on
|
||||
**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded
|
||||
`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools,
|
||||
skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in
|
||||
01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0
|
||||
/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is
|
||||
a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`).
|
||||
|
||||
**Pages:**
|
||||
|
||||
- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the
|
||||
load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the
|
||||
open question (fail-open loader; a registration health check is proposed, **not applied**).
|
||||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for
|
||||
the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak
|
||||
**5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**.
|
||||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in
|
||||
two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire
|
||||
**without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor.
|
||||
- [[systems/conduit]] — 474 dropped registrations and what they cost.
|
||||
- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers.
|
||||
- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up).
|
||||
- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped.
|
||||
|
||||
**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold
|
||||
(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and
|
||||
recorded here so the next audit does not re-litigate it.
|
||||
|
||||
**Still open from earlier windows:** the capture stall has no established cause and no fix applied
|
||||
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
|
||||
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.
|
||||
|
||||
## 2026-11-18 03:xx — 03:00 stocktake (cron)
|
||||
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged vs 11-17), freshness report `raw/stocktake/freshness-20261118.txt` — 9 stale.
|
||||
- Live-vs-page re-verification of stale pages; 7 re-verified with dated corrections:
|
||||
- `runbooks/provider-health.md` — snippets re-executed: CT600 /models 200 (6,501), omni-101 /api/health 200, fallbacks 200.
|
||||
- `runbooks/update-hermes-safely.md` — prerequisites re-verified (v0.23.0, venv + managed runtimes present).
|
||||
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 now v0.23.1 (0 behind origin/main), CT460 v0.23.0; CT255 mattermost running + peer online → Mattermost-timeout claim superseded.
|
||||
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — coverage monitor ok/streak 0; fix holds (script blocked by lifecycle guard; verified via monitor).
|
||||
- `incidents/2026-10-25-gmail-digest-runaway-and-refire.md` — RESOLVED: last_status ok, one clean daily run 11-12→11-17, no refires; delivery_queued cleared.
|
||||
- `incidents/2026-10-16-omniroute-503-resource-pressure.md` — 22 days clean, stays monitoring.
|
||||
- `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` — no unknown rows in retained window; 10-20 log line aged out (rotation noted on page).
|
||||
- Left stale (flagged only, not verifiable this run): `reports/communities-wa-social-media-audit-2026.md` (24d), `incidents/2026-10-24-conduit-call-request-refused.md` (23d, open behavioural).
|
||||
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
|
||||
|
||||
## 2026-11-19 03:xx — 03:00 stocktake (cron)
|
||||
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged), freshness report `raw/stocktake/freshness-20261119.txt` — 3 stale.
|
||||
- All 3 stale pages re-verified against live sources with dated corrections:
|
||||
- `incidents/2026-10-24-conduit-call-request-refused.md` — `conduit-calls.json history` still 0 entries (call path unexercised ~26d); status stays open.
|
||||
- `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions.
|
||||
- `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements.
|
||||
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
|
||||
|
||||
## 2026-11-20 00:5x — nightly audit (three windows: 2026-11-17, 11-18 and 11-19 AWST)
|
||||
|
||||
**Why three windows:** the audit's own slots at **2026-11-18T00:00:08 → 00:01:43** and **2026-11-19T00:00:16 → 00:03:02** both closed `failed`, so **2026-11-17 and 2026-11-18 were never audited**. This run covers all three.
|
||||
|
||||
- **New page — [[incidents/2026-11-18-wiki-audit-credit-outage]]:** `cbfe434a3b5e` (Wiki Audit — Nightly) is at `failure_streak: 2`, both failures on a chained-provider **credit/auth** fault (last clean run 11-17). Both failed slots carry `delivery_outcome: suppressed`, so **no failure notice ever reached Anthony** — the estate's own auditor was the one thing nobody heard about. Its incident row `cbfe43_fa12cca1ad4b` (`failure_type: auth`, opened 2026-11-18 00:01:42) is still `alerted` with **`alerted_at: None`**, the same silent-skip shape as the 11-10 429 incident.
|
||||
- **New page — [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]:** `96eb95ce12f2` (Mnemosyne Watchdog) has failed **every** 15-minute slot since **2026-11-18 20:01:06**, reaching **115 consecutive failures** at **2026-11-20 00:31:16** (`failure_streak: 115`, `last_status: error`), for one reason: **59 unresolved conflicts against its threshold of 50**. A single durable trip, not 115 faults — but each slot delivers a fresh *failure notice* (9 of the 24 deliveries since 11-18 carry `for_failure: 1`), so the noisy channel is the estate's only channel, and every slot re-opens incident `96eb95_98ade422d195`.
|
||||
- **Corrected — [[incidents/2026-11-14-mnemosyne-capture-stall]] and [[systems/mnemosyne]]:** the freeze is **over**. `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** → the stall ran **105 h 52 m** and ended **with no intervention** (no config edit, restart or repair on the capture path). ⚠ **And the "capture is dead" reading was wrong:** `state.db` lists only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` measures the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect, and the monitor is still failing on it (`failure_streak: 2`; 11-19 00:01 was the only clean slot). Title and body reframed accordingly.
|
||||
- **[[systems/scheduled-tasks]] — new Count bullet (11-17 / 11-18 / 11-19):** ledger **173 rows (171 `completed` / 2 `failed`)** / **172 (153/19)** / **176 (78/98)** — the last is the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18: **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (a *content* delivery, `live adapter send failed: Timed out`, `2026-11-19 08:34:13`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window** at the same count (11-18 and 11-19 went unrecorded, see above) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` `error` (2), `96eb95ce12f2` `error` (115), `ea79a2107e36` `error` (2), `08eb86f02d37` `delivery_failed` (0). `cron_incidents` unclosed **5 → 4**. The 03:00 stocktake (`1a8e90ea2271`) ran `completed` on all three days — **11 m 35 s / 23 m 07 s / 24 m 38 s** — so the estate's structural maintenance kept running while both alarm paths were red.
|
||||
- **[[current-state]]** — new Recent Changes section for the 2026-11-17 → 2026-11-20 window. Frontmatter `updated` / `verified_on` bumped to **2026-11-20** across `index.md`, `current-state.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` and the two touched incident pages.
|
||||
- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**.
|
||||
- **Freshness at publish time** (`raw/stocktake/freshness-20261120.txt`): **9 stale / 127 fresh / 3 no-date of 139 pages**. All nine carry the *same* `updated: 2026-10-29` — one 22-day bulk cohort, not nine findings: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. **Not re-verified by this audit** — the live-vs-page re-verification sweep belongs to the 03:00 stocktake (pattern 23), so the cohort is recorded here rather than silently inherited. Five further pages sit at exactly 21d and become the 11-21 stale list: `incidents/2026-10-27-job-radar-budget-exhausted.md`, `incidents/2026-10-28-job-radar-stage1-iteration-cap.md`, `incidents/2026-10-29-conduit-voice-silence.md`, `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md`, `systems/homepage-dashboard.md`.
|
||||
- **Published (patterns 13/30):** `wiki-maintain.sh` regenerated `systems/skills-index.md` (**299 skills / 67 categories**) and rebuilt Quartz — 139 pages parsed, 867 files emitted to `public/`, `hermes-wiki-serve.service` restarted (`systemctl --user is-active` → `active`, listener on `:9120`). Live check after the rebuild: `index`, `current-state`, `systems/mnemosyne`, `systems/scheduled-tasks`, `systems/skills-index` and all three incident pages return **200** on `http://127.0.0.1:9120/<path>`. Both vault conventions synced from the authoring tree with per-file `sha256` matches (8 files × 2 conventions; freshness report included), committed locally — the vault repo has no remote.
|
||||
|
||||
## 2026-11-20 03:0x — Stocktake (cron 1a8e90ea2271): 9-page stale cohort re-verified live
|
||||
|
||||
- Ran `wiki-maintain.sh` (repo script): skills-index regenerated (**299 skills / 67 categories**), freshness report `raw/stocktake/freshness-20261120.txt` → **9 stale / 127 fresh / 3 no-date of 139**. All nine stale pages were the single 2026-10-29 cohort (22d) flagged by the 01:13 audit — treated as one verification sweep (pattern 23), not nine investigations.
|
||||
- **Live probes:** `~/.hermes/config.yaml` browser block; `ss -ltnp` + process list + `cua-driver --version` on CT460; `camofox-browser.service` (`systemctl --user` → active) + `:9377/health` → ok; CT333 via PVE (`ss`, `ps`, `docker ps -a`, `tailscale ip -4`, gateway `/health`, `workspace/cron/jobs.json`); CT201 via PVE (`docker ps -a`, images, Serve status, `_archived-steel/`, `:8085` probe); OmniRoute chat-completions on both nodes (`cliprox/terra`, `free-stack`, `smart-route`); `command-code/gpt-5.6-terra`; `jobs.json` fleet read.
|
||||
- **All 9 pages re-verified** (`updated`/`verified_on` → 2026-11-20) with dated notes: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`.
|
||||
- **Corrections (claims disproved or drifted):**
|
||||
- `agents/nanobot.md`: Docker list drifted — `tender_brattain` → `ecstatic_swartz` (random-name rename class), `quirky_darwin` removed (absent with `-a`), stopped `qdrant` present. Gateway PID now 1256104 (v0.4.1, `/health` ok). **Cron section rewritten from live scheduler**: 11 jobs / 8 enabled; `morning-quota-report`, `daily-agent-briefing`, `evening-quota-report` disabled; `rss-check-proxmox-selfhst` gone; new `nightly-skills-export-sync` (04:00) and `morning-news-podcast` (07:00). Quota pair 07:30/19:00 remains enabled. Scheduler file is `workspace/cron/jobs.json` (the `~/.nanobot/cron/` dir is empty).
|
||||
- `comparisons/camoufox-vs-chrome.md`: struck the stale "9377 not listening on CT460 / service on CT450" claim — `:9377` **is** listening on CT460 (local `camofox-browser.service`, active); ws-server also on `:9222`.
|
||||
- `decisions/rollback-steel-to-camofox.md`: Rationale's "(CT450)" host attribution corrected to **CT460** with strikethrough.
|
||||
- `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27).
|
||||
- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists).
|
||||
- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced.
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
# Wiki freshness audit — 2026-11-16
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (1)
|
||||
|
||||
- [ 22d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||
|
||||
## FRESH (132)
|
||||
|
||||
- [ 5d] CONTRIBUTING.md
|
||||
- [ 8d] agents/hermes-chaos.md
|
||||
- [ 8d] agents/hermes-production.md
|
||||
- [ 5d] agents/index.md
|
||||
- [ 18d] agents/nanobot.md
|
||||
- [ 5d] agents/open-claw.md
|
||||
- [ 12d] agents/qwenpaw.md
|
||||
- [ 5d] agents/zeroclaw.md
|
||||
- [ 18d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 5d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 5d] comparisons/memory-backends.md
|
||||
- [ 5d] comparisons/model-routers.md
|
||||
- [ 18d] comparisons/steel-vs-camoufox.md
|
||||
- [ 5d] concepts/agent-memory.md
|
||||
- [ 18d] concepts/browser-automation.md
|
||||
- [ 5d] concepts/model-routing.md
|
||||
- [ 5d] concepts/persistent-shells.md
|
||||
- [ 5d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 18d] decisions/index.md
|
||||
- [ 5d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 5d] decisions/persistent-shell-enabled.md
|
||||
- [ 18d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 5d] experiments/index.md
|
||||
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 5d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 11d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 12d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 4d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 5d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 4d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 12d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 18d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 21d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 21d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 18d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 1d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 15d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 20d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 6d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 20d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 21d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 21d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 19d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 17d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 17d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 17d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 16d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 17d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 16d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 15d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 15d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 15d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 14d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 13d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 12d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 10d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 10d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 11d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 7d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 11d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 9d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 7d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 5d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 5d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 4d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 3d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 0d] index.md
|
||||
- [ 9d] infrastructure/backups.md
|
||||
- [ 5d] infrastructure/credentials-map.md
|
||||
- [ 9d] infrastructure/docker-services.md
|
||||
- [ 5d] infrastructure/domains-and-tunnels.md
|
||||
- [ 9d] infrastructure/hosts.md
|
||||
- [ 9d] infrastructure/network-map.md
|
||||
- [ 9d] infrastructure/proxmox.md
|
||||
- [ 4d] infrastructure/tailscale.md
|
||||
- [ 12d] queries/retained-research-answers.md
|
||||
- [ 15d] reports/uber-eats/2026-07.md
|
||||
- [ 15d] reports/uber-eats/2026-08.md
|
||||
- [ 15d] reports/uber-eats/2026-09.md
|
||||
- [ 3d] reports/uber-eats/2026-10.md
|
||||
- [ 3d] reports/uber-eats/index.md
|
||||
- [ 5d] runbooks/backup-wiki.md
|
||||
- [ 5d] runbooks/diagnose-docker.md
|
||||
- [ 5d] runbooks/gateway-resource-resilience.md
|
||||
- [ 9d] runbooks/job-radar.md
|
||||
- [ 5d] runbooks/multiplexer-setup.md
|
||||
- [ 21d] runbooks/provider-health.md
|
||||
- [ 18d] runbooks/recover-docker-service.md
|
||||
- [ 5d] runbooks/restart-browser.md
|
||||
- [ 5d] runbooks/restart-hermes.md
|
||||
- [ 5d] runbooks/restore-openrouter.md
|
||||
- [ 5d] runbooks/test-openrouter.md
|
||||
- [ 5d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 21d] runbooks/update-hermes-safely.md
|
||||
- [ 8d] systems/browser-backend.md
|
||||
- [ 5d] systems/cloudflare-tunnel.md
|
||||
- [ 7d] systems/conduit.md
|
||||
- [ 21d] systems/current-profile.md
|
||||
- [ 5d] systems/guanaco.md
|
||||
- [ 5d] systems/headless-server.md
|
||||
- [ 7d] systems/hermes-agent.md
|
||||
- [ 3d] systems/hermes-hq-push.md
|
||||
- [ 5d] systems/hermes-ui.md
|
||||
- [ 12d] systems/home-assistant.md
|
||||
- [ 17d] systems/homepage-dashboard.md
|
||||
- [ 12d] systems/ignis.md
|
||||
- [ 21d] systems/memory-backend.md
|
||||
- [ 14d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 21d] systems/model-providers.md
|
||||
- [ 12d] systems/nanobot.md
|
||||
- [ 5d] systems/nextcloud.md
|
||||
- [ 5d] systems/obsidian.md
|
||||
- [ 6d] systems/omniroute.md
|
||||
- [ 5d] systems/opencode-go.md
|
||||
- [ 5d] systems/openrouter.md
|
||||
- [ 12d] systems/paseo.md
|
||||
- [ 1d] systems/phonon-tts.md
|
||||
- [ 6d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 5d] systems/terminal-backend.md
|
||||
- [ 5d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 1 stale, 132 fresh, 3 no-date of 136 pages
|
||||
@@ -0,0 +1,150 @@
|
||||
# Wiki freshness audit — 2026-11-17
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (7)
|
||||
|
||||
- [ 23d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||
- [ 22d] incidents/2026-10-03-ct450-hermes-update.md (updated 2026-10-26)
|
||||
- [ 22d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md (updated 2026-10-26)
|
||||
- [ 22d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
|
||||
- [ 22d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md (updated 2026-10-26)
|
||||
- [ 22d] runbooks/provider-health.md (updated 2026-10-26)
|
||||
- [ 22d] runbooks/update-hermes-safely.md (updated 2026-10-26)
|
||||
|
||||
## FRESH (127)
|
||||
|
||||
- [ 6d] CONTRIBUTING.md
|
||||
- [ 9d] agents/hermes-chaos.md
|
||||
- [ 9d] agents/hermes-production.md
|
||||
- [ 6d] agents/index.md
|
||||
- [ 19d] agents/nanobot.md
|
||||
- [ 6d] agents/open-claw.md
|
||||
- [ 13d] agents/qwenpaw.md
|
||||
- [ 6d] agents/zeroclaw.md
|
||||
- [ 19d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 6d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 6d] comparisons/memory-backends.md
|
||||
- [ 6d] comparisons/model-routers.md
|
||||
- [ 19d] comparisons/steel-vs-camoufox.md
|
||||
- [ 6d] concepts/agent-memory.md
|
||||
- [ 19d] concepts/browser-automation.md
|
||||
- [ 6d] concepts/model-routing.md
|
||||
- [ 6d] concepts/persistent-shells.md
|
||||
- [ 6d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 19d] decisions/index.md
|
||||
- [ 6d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 6d] decisions/persistent-shell-enabled.md
|
||||
- [ 19d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 6d] experiments/index.md
|
||||
- [ 5d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 5d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 12d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 13d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 6d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 13d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 19d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 19d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 16d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 21d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 21d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 20d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 18d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 18d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 18d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 17d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 18d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 17d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 16d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 16d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 16d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 15d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 14d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 13d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 11d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 11d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 12d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 8d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 12d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 10d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 8d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 0d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 6d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 6d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 5d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 4d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 0d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||
- [ 0d] index.md
|
||||
- [ 10d] infrastructure/backups.md
|
||||
- [ 6d] infrastructure/credentials-map.md
|
||||
- [ 10d] infrastructure/docker-services.md
|
||||
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||
- [ 10d] infrastructure/hosts.md
|
||||
- [ 10d] infrastructure/network-map.md
|
||||
- [ 10d] infrastructure/proxmox.md
|
||||
- [ 5d] infrastructure/tailscale.md
|
||||
- [ 13d] queries/retained-research-answers.md
|
||||
- [ 16d] reports/uber-eats/2026-07.md
|
||||
- [ 16d] reports/uber-eats/2026-08.md
|
||||
- [ 16d] reports/uber-eats/2026-09.md
|
||||
- [ 4d] reports/uber-eats/2026-10.md
|
||||
- [ 4d] reports/uber-eats/index.md
|
||||
- [ 6d] runbooks/backup-wiki.md
|
||||
- [ 6d] runbooks/diagnose-docker.md
|
||||
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||
- [ 10d] runbooks/job-radar.md
|
||||
- [ 6d] runbooks/multiplexer-setup.md
|
||||
- [ 19d] runbooks/recover-docker-service.md
|
||||
- [ 6d] runbooks/restart-browser.md
|
||||
- [ 6d] runbooks/restart-hermes.md
|
||||
- [ 6d] runbooks/restore-openrouter.md
|
||||
- [ 6d] runbooks/test-openrouter.md
|
||||
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 9d] systems/browser-backend.md
|
||||
- [ 6d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 0d] systems/current-profile.md
|
||||
- [ 6d] systems/guanaco.md
|
||||
- [ 6d] systems/headless-server.md
|
||||
- [ 8d] systems/hermes-agent.md
|
||||
- [ 4d] systems/hermes-hq-push.md
|
||||
- [ 6d] systems/hermes-ui.md
|
||||
- [ 13d] systems/home-assistant.md
|
||||
- [ 18d] systems/homepage-dashboard.md
|
||||
- [ 13d] systems/ignis.md
|
||||
- [ 0d] systems/memory-backend.md
|
||||
- [ 15d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 0d] systems/model-providers.md
|
||||
- [ 13d] systems/nanobot.md
|
||||
- [ 6d] systems/nextcloud.md
|
||||
- [ 6d] systems/obsidian.md
|
||||
- [ 7d] systems/omniroute.md
|
||||
- [ 6d] systems/opencode-go.md
|
||||
- [ 6d] systems/openrouter.md
|
||||
- [ 13d] systems/paseo.md
|
||||
- [ 2d] systems/phonon-tts.md
|
||||
- [ 7d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 6d] systems/terminal-backend.md
|
||||
- [ 6d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 7 stale, 127 fresh, 3 no-date of 137 pages
|
||||
@@ -0,0 +1,150 @@
|
||||
# Wiki freshness audit — 2026-11-18
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (2)
|
||||
|
||||
- [ 24d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||
- [ 23d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
|
||||
|
||||
## FRESH (132)
|
||||
|
||||
- [ 7d] CONTRIBUTING.md
|
||||
- [ 10d] agents/hermes-chaos.md
|
||||
- [ 10d] agents/hermes-production.md
|
||||
- [ 7d] agents/index.md
|
||||
- [ 20d] agents/nanobot.md
|
||||
- [ 7d] agents/open-claw.md
|
||||
- [ 14d] agents/qwenpaw.md
|
||||
- [ 7d] agents/zeroclaw.md
|
||||
- [ 20d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 7d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 7d] comparisons/memory-backends.md
|
||||
- [ 7d] comparisons/model-routers.md
|
||||
- [ 20d] comparisons/steel-vs-camoufox.md
|
||||
- [ 7d] concepts/agent-memory.md
|
||||
- [ 20d] concepts/browser-automation.md
|
||||
- [ 7d] concepts/model-routing.md
|
||||
- [ 7d] concepts/persistent-shells.md
|
||||
- [ 7d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 20d] decisions/index.md
|
||||
- [ 7d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 7d] decisions/persistent-shell-enabled.md
|
||||
- [ 20d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 7d] experiments/index.md
|
||||
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 13d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 7d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 14d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 21d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 19d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 19d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 19d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 18d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 19d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 18d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 17d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 17d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 17d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 16d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 15d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 14d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 12d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 12d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 13d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 9d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 13d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 11d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 9d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 7d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 7d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 6d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 5d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 1d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 1d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||
- [ 1d] index.md
|
||||
- [ 11d] infrastructure/backups.md
|
||||
- [ 7d] infrastructure/credentials-map.md
|
||||
- [ 11d] infrastructure/docker-services.md
|
||||
- [ 7d] infrastructure/domains-and-tunnels.md
|
||||
- [ 11d] infrastructure/hosts.md
|
||||
- [ 11d] infrastructure/network-map.md
|
||||
- [ 11d] infrastructure/proxmox.md
|
||||
- [ 6d] infrastructure/tailscale.md
|
||||
- [ 14d] queries/retained-research-answers.md
|
||||
- [ 17d] reports/uber-eats/2026-07.md
|
||||
- [ 17d] reports/uber-eats/2026-08.md
|
||||
- [ 17d] reports/uber-eats/2026-09.md
|
||||
- [ 5d] reports/uber-eats/2026-10.md
|
||||
- [ 5d] reports/uber-eats/index.md
|
||||
- [ 7d] runbooks/backup-wiki.md
|
||||
- [ 7d] runbooks/diagnose-docker.md
|
||||
- [ 7d] runbooks/gateway-resource-resilience.md
|
||||
- [ 11d] runbooks/job-radar.md
|
||||
- [ 7d] runbooks/multiplexer-setup.md
|
||||
- [ 0d] runbooks/provider-health.md
|
||||
- [ 20d] runbooks/recover-docker-service.md
|
||||
- [ 7d] runbooks/restart-browser.md
|
||||
- [ 7d] runbooks/restart-hermes.md
|
||||
- [ 7d] runbooks/restore-openrouter.md
|
||||
- [ 7d] runbooks/test-openrouter.md
|
||||
- [ 7d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 0d] runbooks/update-hermes-safely.md
|
||||
- [ 10d] systems/browser-backend.md
|
||||
- [ 7d] systems/cloudflare-tunnel.md
|
||||
- [ 1d] systems/conduit.md
|
||||
- [ 1d] systems/current-profile.md
|
||||
- [ 7d] systems/guanaco.md
|
||||
- [ 7d] systems/headless-server.md
|
||||
- [ 9d] systems/hermes-agent.md
|
||||
- [ 5d] systems/hermes-hq-push.md
|
||||
- [ 7d] systems/hermes-ui.md
|
||||
- [ 14d] systems/home-assistant.md
|
||||
- [ 19d] systems/homepage-dashboard.md
|
||||
- [ 14d] systems/ignis.md
|
||||
- [ 1d] systems/memory-backend.md
|
||||
- [ 16d] systems/messaging-integrations.md
|
||||
- [ 1d] systems/mnemosyne.md
|
||||
- [ 1d] systems/model-providers.md
|
||||
- [ 14d] systems/nanobot.md
|
||||
- [ 7d] systems/nextcloud.md
|
||||
- [ 7d] systems/obsidian.md
|
||||
- [ 8d] systems/omniroute.md
|
||||
- [ 7d] systems/opencode-go.md
|
||||
- [ 7d] systems/openrouter.md
|
||||
- [ 14d] systems/paseo.md
|
||||
- [ 3d] systems/phonon-tts.md
|
||||
- [ 8d] systems/push-notifications.md
|
||||
- [ 1d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 7d] systems/terminal-backend.md
|
||||
- [ 7d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 2 stale, 132 fresh, 3 no-date of 137 pages
|
||||
@@ -0,0 +1,150 @@
|
||||
# Wiki freshness audit — 2026-11-19
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (134)
|
||||
|
||||
- [ 8d] CONTRIBUTING.md
|
||||
- [ 11d] agents/hermes-chaos.md
|
||||
- [ 11d] agents/hermes-production.md
|
||||
- [ 8d] agents/index.md
|
||||
- [ 21d] agents/nanobot.md
|
||||
- [ 8d] agents/open-claw.md
|
||||
- [ 15d] agents/qwenpaw.md
|
||||
- [ 8d] agents/zeroclaw.md
|
||||
- [ 21d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 8d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 8d] comparisons/memory-backends.md
|
||||
- [ 8d] comparisons/model-routers.md
|
||||
- [ 21d] comparisons/steel-vs-camoufox.md
|
||||
- [ 8d] concepts/agent-memory.md
|
||||
- [ 21d] concepts/browser-automation.md
|
||||
- [ 8d] concepts/model-routing.md
|
||||
- [ 8d] concepts/persistent-shells.md
|
||||
- [ 8d] concepts/tool-calling.md
|
||||
- [ 2d] current-state.md
|
||||
- [ 21d] decisions/index.md
|
||||
- [ 8d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 8d] decisions/persistent-shell-enabled.md
|
||||
- [ 21d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 8d] experiments/index.md
|
||||
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 14d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 8d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 13d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 7d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 4d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 20d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 20d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 20d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 19d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 20d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 19d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 18d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 18d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 18d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 17d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 16d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 15d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 13d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 13d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 14d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 10d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 14d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 12d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 10d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 8d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 8d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 7d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 6d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 2d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 2d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||
- [ 2d] index.md
|
||||
- [ 12d] infrastructure/backups.md
|
||||
- [ 8d] infrastructure/credentials-map.md
|
||||
- [ 12d] infrastructure/docker-services.md
|
||||
- [ 8d] infrastructure/domains-and-tunnels.md
|
||||
- [ 12d] infrastructure/hosts.md
|
||||
- [ 12d] infrastructure/network-map.md
|
||||
- [ 12d] infrastructure/proxmox.md
|
||||
- [ 7d] infrastructure/tailscale.md
|
||||
- [ 15d] queries/retained-research-answers.md
|
||||
- [ 0d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 18d] reports/uber-eats/2026-07.md
|
||||
- [ 18d] reports/uber-eats/2026-08.md
|
||||
- [ 18d] reports/uber-eats/2026-09.md
|
||||
- [ 6d] reports/uber-eats/2026-10.md
|
||||
- [ 6d] reports/uber-eats/index.md
|
||||
- [ 8d] runbooks/backup-wiki.md
|
||||
- [ 8d] runbooks/diagnose-docker.md
|
||||
- [ 8d] runbooks/gateway-resource-resilience.md
|
||||
- [ 12d] runbooks/job-radar.md
|
||||
- [ 8d] runbooks/multiplexer-setup.md
|
||||
- [ 1d] runbooks/provider-health.md
|
||||
- [ 21d] runbooks/recover-docker-service.md
|
||||
- [ 8d] runbooks/restart-browser.md
|
||||
- [ 8d] runbooks/restart-hermes.md
|
||||
- [ 8d] runbooks/restore-openrouter.md
|
||||
- [ 8d] runbooks/test-openrouter.md
|
||||
- [ 8d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 1d] runbooks/update-hermes-safely.md
|
||||
- [ 11d] systems/browser-backend.md
|
||||
- [ 8d] systems/cloudflare-tunnel.md
|
||||
- [ 2d] systems/conduit.md
|
||||
- [ 2d] systems/current-profile.md
|
||||
- [ 8d] systems/guanaco.md
|
||||
- [ 8d] systems/headless-server.md
|
||||
- [ 10d] systems/hermes-agent.md
|
||||
- [ 6d] systems/hermes-hq-push.md
|
||||
- [ 8d] systems/hermes-ui.md
|
||||
- [ 15d] systems/home-assistant.md
|
||||
- [ 20d] systems/homepage-dashboard.md
|
||||
- [ 15d] systems/ignis.md
|
||||
- [ 2d] systems/memory-backend.md
|
||||
- [ 17d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 2d] systems/model-providers.md
|
||||
- [ 15d] systems/nanobot.md
|
||||
- [ 8d] systems/nextcloud.md
|
||||
- [ 8d] systems/obsidian.md
|
||||
- [ 9d] systems/omniroute.md
|
||||
- [ 8d] systems/opencode-go.md
|
||||
- [ 8d] systems/openrouter.md
|
||||
- [ 15d] systems/paseo.md
|
||||
- [ 4d] systems/phonon-tts.md
|
||||
- [ 9d] systems/push-notifications.md
|
||||
- [ 2d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 8d] systems/terminal-backend.md
|
||||
- [ 8d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 134 fresh, 3 no-date of 137 pages
|
||||
@@ -0,0 +1,152 @@
|
||||
# Wiki freshness audit — 2026-11-20
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (136)
|
||||
|
||||
- [ 9d] CONTRIBUTING.md
|
||||
- [ 12d] agents/hermes-chaos.md
|
||||
- [ 12d] agents/hermes-production.md
|
||||
- [ 9d] agents/index.md
|
||||
- [ 0d] agents/nanobot.md
|
||||
- [ 9d] agents/open-claw.md
|
||||
- [ 16d] agents/qwenpaw.md
|
||||
- [ 9d] agents/zeroclaw.md
|
||||
- [ 0d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 9d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 9d] comparisons/memory-backends.md
|
||||
- [ 9d] comparisons/model-routers.md
|
||||
- [ 0d] comparisons/steel-vs-camoufox.md
|
||||
- [ 9d] concepts/agent-memory.md
|
||||
- [ 0d] concepts/browser-automation.md
|
||||
- [ 9d] concepts/model-routing.md
|
||||
- [ 9d] concepts/persistent-shells.md
|
||||
- [ 9d] concepts/tool-calling.md
|
||||
- [ 0d] current-state.md
|
||||
- [ 0d] decisions/index.md
|
||||
- [ 9d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 9d] decisions/persistent-shell-enabled.md
|
||||
- [ 0d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 9d] experiments/index.md
|
||||
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 13d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 15d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 9d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 8d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 2d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 10d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 2d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 9d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 1d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 21d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 21d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 21d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 20d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 21d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 20d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 19d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 19d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 19d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 18d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 17d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 16d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 14d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 14d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 15d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 11d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 15d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 13d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 11d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 3d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||
- [ 9d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||
- [ 9d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||
- [ 8d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||
- [ 7d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||
- [ 3d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||
- [ 0d] incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
|
||||
- [ 0d] incidents/2026-11-18-wiki-audit-credit-outage.md
|
||||
- [ 0d] index.md
|
||||
- [ 13d] infrastructure/backups.md
|
||||
- [ 9d] infrastructure/credentials-map.md
|
||||
- [ 13d] infrastructure/docker-services.md
|
||||
- [ 9d] infrastructure/domains-and-tunnels.md
|
||||
- [ 13d] infrastructure/hosts.md
|
||||
- [ 13d] infrastructure/network-map.md
|
||||
- [ 13d] infrastructure/proxmox.md
|
||||
- [ 8d] infrastructure/tailscale.md
|
||||
- [ 16d] queries/retained-research-answers.md
|
||||
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 19d] reports/uber-eats/2026-07.md
|
||||
- [ 19d] reports/uber-eats/2026-08.md
|
||||
- [ 19d] reports/uber-eats/2026-09.md
|
||||
- [ 7d] reports/uber-eats/2026-10.md
|
||||
- [ 7d] reports/uber-eats/index.md
|
||||
- [ 9d] runbooks/backup-wiki.md
|
||||
- [ 9d] runbooks/diagnose-docker.md
|
||||
- [ 9d] runbooks/gateway-resource-resilience.md
|
||||
- [ 13d] runbooks/job-radar.md
|
||||
- [ 9d] runbooks/multiplexer-setup.md
|
||||
- [ 2d] runbooks/provider-health.md
|
||||
- [ 0d] runbooks/recover-docker-service.md
|
||||
- [ 9d] runbooks/restart-browser.md
|
||||
- [ 9d] runbooks/restart-hermes.md
|
||||
- [ 9d] runbooks/restore-openrouter.md
|
||||
- [ 9d] runbooks/test-openrouter.md
|
||||
- [ 9d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 2d] runbooks/update-hermes-safely.md
|
||||
- [ 12d] systems/browser-backend.md
|
||||
- [ 9d] systems/cloudflare-tunnel.md
|
||||
- [ 3d] systems/conduit.md
|
||||
- [ 3d] systems/current-profile.md
|
||||
- [ 9d] systems/guanaco.md
|
||||
- [ 9d] systems/headless-server.md
|
||||
- [ 11d] systems/hermes-agent.md
|
||||
- [ 7d] systems/hermes-hq-push.md
|
||||
- [ 9d] systems/hermes-ui.md
|
||||
- [ 16d] systems/home-assistant.md
|
||||
- [ 21d] systems/homepage-dashboard.md
|
||||
- [ 16d] systems/ignis.md
|
||||
- [ 3d] systems/memory-backend.md
|
||||
- [ 18d] systems/messaging-integrations.md
|
||||
- [ 0d] systems/mnemosyne.md
|
||||
- [ 3d] systems/model-providers.md
|
||||
- [ 16d] systems/nanobot.md
|
||||
- [ 9d] systems/nextcloud.md
|
||||
- [ 9d] systems/obsidian.md
|
||||
- [ 10d] systems/omniroute.md
|
||||
- [ 9d] systems/opencode-go.md
|
||||
- [ 9d] systems/openrouter.md
|
||||
- [ 16d] systems/paseo.md
|
||||
- [ 5d] systems/phonon-tts.md
|
||||
- [ 10d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 9d] systems/terminal-backend.md
|
||||
- [ 9d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 136 fresh, 3 no-date of 139 pages
|
||||
@@ -3,8 +3,8 @@ title: Department of Communities WA Social Media Audit and First-Day Brief
|
||||
type: report
|
||||
status: active
|
||||
created: 2026-08-28
|
||||
updated: 2026-10-25
|
||||
verified_on: 2026-10-25
|
||||
updated: 2026-11-19
|
||||
verified_on: 2026-11-19
|
||||
review_after: 2026-11-26
|
||||
confidence: medium
|
||||
tags: [content, research, department-of-communities, social-media]
|
||||
@@ -37,6 +37,13 @@ point-in-time public-source figures from the 2026-08-28 audit and remain as reco
|
||||
not re-counted. The `review_after` date of 2026-11-26 stands for a full refresh against native
|
||||
analytics.
|
||||
|
||||
**Re-verified 2026-11-19 (stocktake).** Live check of wa.gov.au/organisation/department-of-communities
|
||||
(page last updated 18 November 2026): the remit text is **verbatim-identical** to the quote above,
|
||||
all **five Ministers** are still listed (McGurk, Winton, Beazley, Stojkovski, Swinbourn), and the
|
||||
announcement flow (child-safety package 18 Nov, Seniors Card guide 12 Nov, Seniors Week 11 Nov)
|
||||
confirms the recorded content pillars remain active topics. Structural claims of this page all
|
||||
still hold.
|
||||
|
||||
## Strategic baseline
|
||||
|
||||
- The Department's current remit spans child protection, community services, homelessness, prevention of family and domestic violence, disability services, early childhood education and care regulation, women's interests, youth, seniors and ageing, carers, and volunteering.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Provider Health Check
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, providers, health, routing]
|
||||
@@ -58,6 +58,8 @@ with urllib.request.urlopen(req, timeout=5) as r:
|
||||
```
|
||||
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
||||
|
||||
> **Re-executed 2026-11-18:** step-3 snippet OK — CT600 `/models` → 200, **6,501 models** (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 `/api/health` → 200; `omni-101` `/api/health` → 200; fallback endpoints `opencode.ai` / `inference-api.nousresearch.com` / `ollama.com` all → 200.
|
||||
|
||||
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
||||
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
||||
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Recover a Docker service
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-29
|
||||
verified_on: 2026-10-29
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
tags: [runbook, docker, recovery]
|
||||
sources: []
|
||||
confidence: high
|
||||
@@ -20,6 +20,11 @@ Container shows healthy after recovery probe reboots.
|
||||
## Last successfully used
|
||||
2026-07-22
|
||||
|
||||
> **Re-verified 2026-11-20 (stocktake, procedure only).** `docker ps` as `hermes` on CT460 still
|
||||
> returns permission denied (`/var/run/docker.sock` is `root:docker 0660`) — sudo or SSH to the
|
||||
> owning host remains required. CT201 homepage degradation **persists**: `:8085` answers **502**
|
||||
> locally, `homepage-manager` up but no dashboard container on 8082. Steel remains fully removed
|
||||
> (re-probed CT201/CT333: no containers, no images, no routes).
|
||||
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
|
||||
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
|
||||
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Update Hermes safely
|
||||
type: runbook
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-18
|
||||
verified_on: 2026-11-18
|
||||
last_tested: 2026-10-26
|
||||
confidence: high
|
||||
tags: [runbook, hermes, update]
|
||||
@@ -28,6 +28,8 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
|
||||
- Available disk space in Hermes install and backup paths
|
||||
- No active Hermes chat/webui shell issues before starting
|
||||
|
||||
> **Re-verified 2026-11-18:** prerequisites hold on CT460 — `hermes --version` → `v0.23.0` from the git install at `/home/hermes/.hermes/hermes-agent`, legacy `venv/` present, three managed runtime venvs present under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. (CT450 currently runs `v0.23.1` — see [[incidents/2026-10-03-ct450-hermes-update]].)
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Create a backup first
|
||||
|
||||
+15
-2
@@ -3,8 +3,8 @@ title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-11-09
|
||||
verified_on: 2026-11-09
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-12-08
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
||||
@@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push
|
||||
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||
|
||||
### 2026-11-15 — 474 registrations dropped across eight failed loads
|
||||
|
||||
`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after
|
||||
10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called
|
||||
register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events
|
||||
(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations,
|
||||
so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the
|
||||
caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out
|
||||
**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts:
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side:
|
||||
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||
|
||||
|
||||
## Open items
|
||||
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
||||
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
||||
|
||||
+16
-16
@@ -3,9 +3,9 @@ title: Current Hermes Profile
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
review_after: 2026-11-09
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-12-01
|
||||
confidence: high
|
||||
tags: [system, hermes, profile]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
||||
@@ -16,9 +16,9 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
||||
## Active Profile
|
||||
- **Name:** `default`
|
||||
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
||||
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
|
||||
- **Other profiles present:** none — `~/.hermes/profiles/` holds only a `.deleted/` marker. The `ops` (Tusk elephant) profile was **deleted 2026-10-11**; two chats still routed to it afterwards, so expect stray references. Other profiles do **not** affect this session; see the profile-isolation note below.
|
||||
|
||||
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
|
||||
## Key Settings (re-verified against live config 2026-11-17)
|
||||
|
||||
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
|
||||
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
|
||||
@@ -38,24 +38,24 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
||||
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
|
||||
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
|
||||
credential** — a fallover would try it first.
|
||||
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
|
||||
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
|
||||
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
|
||||
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
|
||||
- **⚠ Cron pins (re-verified live from `jobs.json` 2026-11-17).** **21 of the 26 enabled jobs pin OmniRoute
|
||||
explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1); the other 5 enabled jobs carry no
|
||||
pin (no-agent/script jobs) and follow the global default, which is OmniRoute — the fleet runs a **single
|
||||
upstream path**. → [[systems/scheduled-tasks]]
|
||||
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
|
||||
`model.fallbacks` does not exist.
|
||||
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
|
||||
[[systems/model-providers]].
|
||||
- ⚠ The `custom:omni` provider key is **no longer configured** and **no jobs name it any more** (verified from
|
||||
`jobs.json` 2026-11-17; two jobs still did on 2026-10-26) — see [[systems/model-providers]].
|
||||
|
||||
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
|
||||
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
|
||||
### Other settings (re-verified live 2026-11-17)
|
||||
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13. A ninth slot, **`pulse_memory`**, now points at `nous` / `anthropic/claude-haiku-4.5` — i.e. the **dead nous credential**. → [[systems/model-providers]]
|
||||
- **Switch persistence:** `persist_switch_by_default: false`
|
||||
- **Timezone:** `Australia/Perth`
|
||||
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
||||
- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
|
||||
- **Config version:** `_config_version: 46`; config file ≈514 KB / ≈13,200 lines
|
||||
- **MCP servers:** 5 configured
|
||||
- **Plugins enabled:** 11
|
||||
- **Config version:** `_config_version: 52` (config file ≈514 KB / ≈13,200 lines as of 2026-10-26)
|
||||
- **MCP servers:** 6 configured (`Not human search`, `asana` (disabled), `cua-driver`, `hound`, `mcp-portal`, `approved_shared_memory`)
|
||||
- **Plugins enabled:** 15
|
||||
- **Pet:** configured slug `rhino`, but `enabled: false` (render mode `unicode`, scale 0.55)
|
||||
- **Skin:** `slate`
|
||||
- **Personality:** empty string in config — the active persona comes from `SOUL.md`, not this key
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Memory Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, agent]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -34,7 +34,7 @@ Persist and recall agent memory across sessions.
|
||||
- **Sleep threshold:** 50 turns
|
||||
- **Nudge interval:** 10
|
||||
- **Ignore patterns:** none set
|
||||
- **Live totals (2026-10-26):** 2,350 memories — 842 working / 1,090 episodic / 418 `memoria_facts`
|
||||
- **Live totals (2026-11-17, sqlite):** 2,901 memories — 1,130 working / 1,249 episodic / 522 `memoria_facts`
|
||||
|
||||
## Storage Location
|
||||
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
||||
@@ -48,7 +48,7 @@ Persist and recall agent memory across sessions.
|
||||
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||
- **Embedding coverage re-verified 2026-10-26** — `check_embedding_coverage.py` reports 1,899 / 2,350 (80.8 %) embedded; **working 815 / 842 (96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)**. Both tiers are **above the 95 % reindex threshold**, so no reindex was triggered. `fastembed` 0.8.1 is present in **both** managed runtimes and 0.8.0 in the legacy venv; `FASTEMBED_CACHE_PATH` is pinned. The 2026-10-03 follow-up (assert `fastembed` in *all* managed runtimes) is now satisfied.
|
||||
- **Embedding coverage re-verified 2026-11-17** — `check_embedding_coverage.py` reports 2,379 / 2,901 (82.0 %) embedded; **working 1,130 / 1,130 (100 %)** and **episodic 1,249 / 1,249 (100 %)** (the gap is `memoria_facts`, unvectorized by design). Both tiers are **above the 95 % reindex threshold**, so no reindex is needed. Earlier (2026-10-26) check: 1,899 / 2,350 (80.8 %), working 96.8 %, episodic 99.4 %.
|
||||
|
||||
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -3,8 +3,8 @@ title: Model Providers
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
confidence: high
|
||||
tags: [model, provider, inference, routing]
|
||||
sources: [live-probe, config, web-search]
|
||||
@@ -20,7 +20,7 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
|
||||
|
||||
| Provider | URL | Models | Role |
|
||||
|---|---|---|---|
|
||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
|
||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 21 of 26 enabled jobs pin it (2026-11-17) |
|
||||
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
||||
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
||||
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
||||
@@ -53,13 +53,15 @@ Provider names that appear in job configs but not in `custom_providers` do **not
|
||||
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
||||
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
||||
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
||||
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
|
||||
- **21 of 26 enabled scheduled jobs pin OmniRoute explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1; verified from live `jobs.json` 2026-11-17). The remaining 5 enabled jobs carry no pin (no-agent/script jobs) and follow the global default, which is **also** OmniRoute — the fleet runs a **single upstream path**. The two jobs that still named the removed `custom:omni` key on 2026-10-26 no longer do. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Auxiliary Role Routing (re-verified live 2026-10-26)
|
||||
## Auxiliary Role Routing (re-verified live 2026-11-17)
|
||||
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
|
||||
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
|
||||
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
|
||||
**Compression timeout 90 s; vision timeout 180 s.**
|
||||
**Compression timeout 90 s; vision timeout 180 s.** A ninth slot, **`pulse_memory`**, is
|
||||
**`nous` / `anthropic/claude-haiku-4.5`** — i.e. it sits on the **dead `nous` credential**; a pulse-memory call
|
||||
that hits it will fail until the slot or the credential is fixed.
|
||||
|
||||
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
|
||||
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
review_after: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
review_after: 2026-11-21
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,17 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-20 00:5x (three-window audit — 2026-11-17, 11-18 and 11-19; the audits slotted at 11-18 00:00 and 11-19 00:00 both died, so 11-17 and 11-18 had never been audited; live `jobs.json` + `executions.db` + `deliveries.db` + `cron_incidents` + `mnemosyne.db`).** Ledger: **11-17 — 173 rows, 171 `completed` / 2 `failed`** (both `ea79a2107e36` *Mnemosyne Health Monitor*, `00:01` and `12:01`); **11-18 — 172 rows, 153 / 19** (this audit `00:00:08 → 00:01:43`, the monitor twice, and **16** *Mnemosyne Watchdog* slots from `20:01:06`); **11-19 — 176 rows, 78 / 98** — **the worst day in the retained window** (audit `00:00:16 → 00:03:02`, monitor `12:01`, **96** watchdog slots); **11-20 so far — 8 rows, 3 `completed` / 4 `failed` / 1 `running`** (this audit). Deliveries since 11-18: **24 — 23 `delivered` (9 `for_failure: 1`) / 1 `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window at the same count** (following the 11th, recorded on 11-17; the 11-18 and 11-19 windows went unrecorded because both audits died) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` Wiki Audit `error`, `failure_streak` **2** (last clean run 11-17), `96eb95ce12f2` Watchdog `error`, streak **115**, `ea79a2107e36` Health Monitor `error`, streak **2**, `08eb86f02d37` Gmail Action Digest `delivery_failed`, streak **0**. **Two of the three jobs that keep the fleet safe are themselves red — and the 03:00 stocktake is not:** `1a8e90ea2271` ran `completed` on 11-17 (**11 m 35 s**), 11-18 (**23 m 07 s**) and 11-19 (**24 m 38 s**), so the estate's structural maintenance kept running while both alarm paths failed. `cron_incidents` unclosed **5 → 4**, none closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, **`alerted_at: None`** — the audit's silent skip), `96eb95_98ade422d195` (*script*, 2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16), `ea79a2_f2be2db3bfce` (2026-11-19 12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36) — both monitor rows on the same `STALE MEMORY PIPELINE` metric that is now measuring a *quiet estate* rather than a broken capture (see [[systems/mnemosyne]]). Cadence and pins unchanged; this audit still runs `custom:omni-101` / `deepseek-paid-smart`, and the Mnemosyne Watchdog's model/provider fields remain unset (inherits global). Full detail: [[incidents/2026-11-18-wiki-audit-credit-outage]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]].
|
||||
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
|
||||
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
|
||||
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
|
||||
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
|
||||
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
|
||||
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
|
||||
failures and **0** hook timeouts, against 52 and 13 the day before — see
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]].
|
||||
|
||||
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-15
|
||||
verified_on: 2026-11-15
|
||||
updated: 2026-11-20
|
||||
verified_on: 2026-11-20
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user