Compare commits
12
Commits
2d439145bf
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f603c29d17 | ||
|
|
8bf0a37022 | ||
|
|
226767f3ec | ||
|
|
16f2c878c0 | ||
|
|
98610466cc | ||
|
|
4574c4ebc3 | ||
|
|
e5612c8a8d | ||
|
|
4eee02b9a2 | ||
|
|
9f80926dda | ||
|
|
402d8546e8 | ||
|
|
646be98c31 | ||
|
|
82876abdfb |
+40
-15
@@ -3,8 +3,8 @@ title: Nanobot
|
|||||||
type: agent
|
type: agent
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [agent, nanobot, homelab]
|
tags: [agent, nanobot, homelab]
|
||||||
sources: [live-probe, runtime-config]
|
sources: [live-probe, runtime-config]
|
||||||
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
|||||||
- **Workspace:** `/home/nano/.nanobot/workspace`
|
- **Workspace:** `/home/nano/.nanobot/workspace`
|
||||||
|
|
||||||
## Current state
|
## Current state
|
||||||
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening.
|
- **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
|
||||||
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
|
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
|
||||||
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
|
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
|
||||||
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
|
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
|
||||||
@@ -38,22 +38,31 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
|
|||||||
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
|
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
|
||||||
|
|
||||||
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
|
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
|
||||||
`gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`.
|
~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
|
||||||
|
|
||||||
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
|
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
|
||||||
|
|
||||||
## Cron jobs
|
## Cron jobs
|
||||||
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded:
|
**Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
|
||||||
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST
|
(all schedules Australia/Perth):
|
||||||
- `morning-quota-report` — daily at 07:30 AWST
|
|
||||||
- `daily-quota-730am` — daily at 07:30 AWST
|
| Job | Schedule | Enabled |
|
||||||
- `vault-daily-seed` — daily at 08:00 AWST
|
|---|---|---|
|
||||||
- `daily-agent-briefing` — daily at 08:15 AWST
|
| `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
|
||||||
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST
|
| `nightly-skills-export-sync` | daily 04:00 | ✅ |
|
||||||
- `daily-quota-7pm` — daily at 19:00 AWST
|
| `morning-news-podcast` | daily 07:00 | ✅ |
|
||||||
- `evening-quota-report` — daily at 21:00 AWST
|
| `daily-quota-730am` | daily 07:30 | ✅ |
|
||||||
- `heartbeat` — every 30m
|
| `morning-quota-report` | daily 07:30 | ❌ disabled |
|
||||||
- `dream` — every 2h (memory consolidation)
|
| `vault-daily-seed` | daily 08:00 | ✅ |
|
||||||
|
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
|
||||||
|
| `daily-quota-7pm` | daily 19:00 | ✅ |
|
||||||
|
| `evening-quota-report` | daily 21:00 | ❌ disabled |
|
||||||
|
| `heartbeat` | every 30m | ✅ |
|
||||||
|
| `dream` | every 2h (memory consolidation) | ✅ |
|
||||||
|
|
||||||
|
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
|
||||||
|
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
|
||||||
|
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
|
||||||
|
|
||||||
## Skills
|
## Skills
|
||||||
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
|
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
|
||||||
@@ -73,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
|
|||||||
- [[runbooks/restart-hermes]]
|
- [[runbooks/restart-hermes]]
|
||||||
- [[runbooks/nanobot-restart]]
|
- [[runbooks/nanobot-restart]]
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
|
||||||
|
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
|
||||||
|
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
|
||||||
|
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
|
||||||
|
> agentmail-daemon and readlater (:3210) processes all still run.
|
||||||
|
>
|
||||||
|
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
|
||||||
|
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
|
||||||
|
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
|
||||||
|
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
|
||||||
|
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
|
||||||
|
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
|
||||||
|
>
|
||||||
|
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
|
||||||
|
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
|
||||||
|
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
|
||||||
## Related
|
## Related
|
||||||
- [[comparisons/hermes-vs-nanobot]]
|
- [[comparisons/hermes-vs-nanobot]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [comparison, browser]
|
tags: [comparison, browser]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -46,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
|
|||||||
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
||||||
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
|
||||||
|
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
|
||||||
|
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
|
||||||
|
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
|
||||||
|
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
|
||||||
|
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
|
||||||
|
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
|
||||||
|
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
|
||||||
|
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
|
||||||
|
> No Chrome/Chromium process found in this probe either.
|
||||||
### Chrome (not found in use)
|
### Chrome (not found in use)
|
||||||
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
|
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
|
||||||
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-29
|
created: 2026-07-29
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, browser, steel, camofox, decision]
|
tags: [comparison, browser, steel, camofox, decision]
|
||||||
sources: [live-probe, session-history]
|
sources: [live-probe, session-history]
|
||||||
@@ -77,6 +77,13 @@ the CT201 and CT333 deployments torn down — so returning to Steel means redepl
|
|||||||
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
|
||||||
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
|
||||||
|
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
|
||||||
|
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
|
||||||
|
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
|
||||||
|
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
|
||||||
|
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
|
||||||
|
> `cloud_provider: camofox`. All removal claims **hold**.
|
||||||
## Related
|
## Related
|
||||||
- [[systems/browser-backend]]
|
- [[systems/browser-backend]]
|
||||||
- [[concepts/browser-automation]]
|
- [[concepts/browser-automation]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Browser Automation
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, browser, camofox, steel]
|
tags: [concept, browser, camofox, steel]
|
||||||
sources: [live-probe, session-history]
|
sources: [live-probe, session-history]
|
||||||
@@ -33,6 +33,10 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
|
|||||||
|
|
||||||
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
|
||||||
|
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
|
||||||
|
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
|
||||||
|
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
|
||||||
## Related
|
## Related
|
||||||
- [[systems/browser-backend]]
|
- [[systems/browser-backend]]
|
||||||
- [[comparisons/steel-vs-camoufox]]
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
|
|||||||
+77
-4
@@ -3,16 +3,89 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-20
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-20
|
||||||
review_after: 2026-11-15
|
review_after: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-11-15
|
Updated: 2026-11-20
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-17 → 2026-11-20, three-window audit; live probes 2026-11-20 00:0x – 00:5x AWST)
|
||||||
|
|
||||||
|
**The estate's memory freeze ended by itself, its two alarm paths are still red, and the audits that would have reported either had silently died twice.**
|
||||||
|
|
||||||
|
- **Capture stall: over, unassisted.** `working_memory` **1130 → 1131**, `episodic_memory` **1249 → 1250**, newest write **`2026-11-18 19:25:03` AWST** — a **105 h 52 m** break that ended with no config change, restart or repair. ⚠ **And the "capture dead" reading was wrong:** `state.db` holds only **three** non-cron sessions since 11-14 00:00 — two `desktop` on **11-14 08:35 / 08:52** (both *before* the 09:33 boundary) and the **11-18 19:18** `desktop` whose turn *is* the captured row — so **every interactive session in the window was captured**. `mnemosyne_freshness_gate.py` reads the age of the newest non-`cron`/`subagent` message, so **silence and failure read identically**; that ambiguity is the real defect. The monitor is still failing on it (`failure_streak: 2`; slots 11-18 00:01 and 12:01, 11-19 12:01, 11-20 00:01 — 11-19 00:01 alone passed). → [[incidents/2026-11-14-mnemosyne-capture-stall]], [[systems/mnemosyne]]
|
||||||
|
- **The nightly audit missed two nights.** `cbfe434a3b5e` failed on **11-18** and **11-19** on a chained-provider credit/auth fault; **both deliveries `suppressed`**, and its incident row is still `alerted` with **`alerted_at: None`** → [[incidents/2026-11-18-wiki-audit-credit-outage]]. **This run covers 11-17, 11-18 and 11-19.**
|
||||||
|
- **The watchdog is in a threshold storm.** `96eb95ce12f2` failed **115 consecutive** 15-minute slots from **2026-11-18 20:01:06** to **2026-11-20 00:31:16** on **59 unresolved conflicts vs a threshold of 50** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]
|
||||||
|
- **Cron ledger.** 11-17 **173 rows (171 `completed` / 2 `failed`)**; 11-18 **172 (153/19)**; 11-19 **176 (78/98)** — the worst day in the retained window; 11-20 so far **8 (3 / 4 / 1 `running`)**. Deliveries since 11-18 **24 — 23 `delivered`** (9 `for_failure: 1`) / **1 `failed`** (content, Telegram timeout `2026-11-19 08:34:13`). Fleet **29 (26 enabled / 3 disabled)**, a twelfth consecutive window; non-clean `last_status` **1 → 4**; unclosed `cron_incidents` **5 → 4**. The **03:00 stocktake ran `completed` on all three days** (11 m 35 s / 23 m 07 s / 24 m 38 s).
|
||||||
|
- **Mnemosyne consolidation still zero-item:** newest `consolidation_log` row `2026-11-19T16:20:14+00:00`, `items_consolidated: 0`, **199 of the last 200 cycles zero-item** (was 120 on 11-17). Zero-item-streak threshold still **proposed, not applied**.
|
||||||
|
- **Pages touched:** [[incidents/2026-11-18-wiki-audit-credit-outage]] (new), [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]] (new), [[incidents/2026-11-14-mnemosyne-capture-stall]] (new re-read + corrected title), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[index]], `log.md` — frontmatter dates → **2026-11-20**.
|
||||||
|
|
||||||
|
**Previous window:**
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-16, nightly audit; live probes 2026-11-16 23:59 – 2026-11-17 00:27 AWST)
|
||||||
|
|
||||||
|
**The memory pipeline is still dead at 62 h, and the 11-15 record needed a correction: a plugin
|
||||||
|
load-timeout storm nobody had logged.**
|
||||||
|
|
||||||
|
- **Capture stall, third read — unchanged in every respect but the counters.** `working_memory` **1130**
|
||||||
|
/ `episodic_memory` **1249**, newest write `2026-11-14 09:32:59` AWST → **62 h 54 m**. Unclosed
|
||||||
|
`alerted` incidents **3 → 5** (143 rows total); monitor `failure_streak` **5**; consolidator zero-item
|
||||||
|
streak **120** cycles. No cause, no fix, `status: active` →
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]
|
||||||
|
- ⚠ **New page — [[incidents/2026-11-15-plugin-load-timeout-storm]].** 11-15 carried **52** failed plugin
|
||||||
|
loads and **519** discarded registrations, `conduit_push` alone losing **474** (its tools and hooks were
|
||||||
|
absent for those processes), plus **11** `pre_tool_call` and **2** `pre_llm_call` hook timeouts and
|
||||||
|
**10** refused tool calls. The 11-15 audit recorded **none** of it. **Zero on 11-16 and 11-17** — it
|
||||||
|
cleared with no fix applied. The companion
|
||||||
|
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] gained a Recurrence 2 section; load average
|
||||||
|
is now **explicitly not** a sufficient predictor (11-17 read 6.19 vs 12.08 on 11-14, no recurrence, and
|
||||||
|
a *larger* `state.db`).
|
||||||
|
- **Cron fleet, 11th window, flat at 29 jobs** (26 enabled / 3 disabled). 11-16 ledger **175 rows —
|
||||||
|
173 `completed` / 2 `failed`**, both the health monitor; deliveries **11/11 `delivered`**. 11-17 so far:
|
||||||
|
5 / 1 / 1. The 11-15 watchdog deps-guard 600 s timeout **did not recur**.
|
||||||
|
- **Quiet where it matters, growing where it doesn't.** 11-16 ran **16 cron sessions, zero interactive** —
|
||||||
|
nothing human-side to document. `state.db` **2,093,998,080 B (2.09 GB)**, WAL **40 MB**, `/` at **88 %**,
|
||||||
|
load **6.19 / 6.64 / 7.48**, uptime 6 d 2 h (host up since 2026-11-10).
|
||||||
|
- **Freshness:** 1 stale page by design — `reports/communities-wa-social-media-audit-2026` (23 d at
|
||||||
|
11-17); it is a static deliverable with `review_after` 2026-11-26, so it was left unbumped.
|
||||||
|
|
||||||
|
|
||||||
|
## Recent Changes (2026-11-15, nightly audit; live probes 2026-11-15 23:59 – 2026-11-16 00:2x AWST)
|
||||||
|
|
||||||
|
- ⚠ **Memory capture is still dead — 38 h 46 m — and the freeze is total, not degraded.**
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]] re-read: `working_memory` **1130** /
|
||||||
|
`episodic_memory` **1249**, *the same two counts as every earlier read of the stall*; newest
|
||||||
|
`working_memory` write still the `09:32:59 AWST` (01:32:59 UTC) `[MODEL_REFRESH_PROPOSAL]` maintenance
|
||||||
|
row, newest *user-turn* capture `2026-11-14 09:04:19 AWST`. Root cause still not established.
|
||||||
|
- **The monitor's `≈4.64 h` figure is not a duration — decoded against the gate's own source.**
|
||||||
|
`mnemosyne_freshness_gate.py`'s `newest_user_message()` takes the newest `role='user'` message whose
|
||||||
|
session source is not `cron`/`subagent` and prints *its* age. On this box those are the **Pulse**
|
||||||
|
turns on `source: desktop` (~6 h cadence), and the `0 */12` slots sit ~4.6 h after one — which is why
|
||||||
|
all three alerts read alike. **Do not read it as the stall length.**
|
||||||
|
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows) — all `ea79a2107e36`, all
|
||||||
|
`failure_type: auth`, all `alerted`, each `first_seen_at == last_seen_at`. The signature embeds the
|
||||||
|
elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), so **one durable
|
||||||
|
fault mints a fresh unclosed row every slot** and nothing escalates. **Fix proposed, not applied.**
|
||||||
|
- **The consolidator runs and does nothing — by design.** Latest `consolidation_log` row **4696** at
|
||||||
|
`2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview`
|
||||||
|
`No-op consolidation cycle — timestamp updated for watchdog`; **~74 consecutive zero-item cycles**
|
||||||
|
from `2026-11-14T02:04:42+00:00`. It bumps its own timestamp, so an age-only gate reads "fresh".
|
||||||
|
- **Fleet: 2026-11-15 — 168 rows, 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`**
|
||||||
|
(3 `for_failure`). The three: the Health Monitor at `00:00:53.928` and `12:01:24.719`, plus — **new
|
||||||
|
failure mode** — the Watchdog `96eb95ce12f2` at `11:31:20.699`, `finished_at 11:49:10.534610` =
|
||||||
|
**17 m 49.8 s**, `runtime deps guard … timed out after 600 seconds`; the `11:45` slot was **skipped**,
|
||||||
|
`12:01:25` completed. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`.**
|
||||||
|
- **Fleet 29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window unchanged**; only non-clean
|
||||||
|
row is `ea79a2107e36` (`error`, `failure_streak` **3**).
|
||||||
|
- **No interactive work on 11-15.** All **9** sessions started that day are `source: cron` — nothing
|
||||||
|
human-side for the wiki to have missed.
|
||||||
|
- Freshness (03:00 stocktake, `raw/stocktake/freshness-20261115.txt`): **0 stale / 133 fresh / 3
|
||||||
|
no-date of 136** pages.
|
||||||
|
|
||||||
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
## Recent Changes (2026-11-14, nightly audit; live probes 2026-11-14 23:59 - 2026-11-15 00:4x AWST)
|
||||||
|
|
||||||
|
|||||||
+6
-2
@@ -3,8 +3,8 @@ title: Decisions
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [decisions]
|
tags: [decisions]
|
||||||
---
|
---
|
||||||
@@ -31,6 +31,10 @@ Stable strategic and config decisions, with evidence and status.
|
|||||||
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
|
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
|
||||||
> backend.
|
> backend.
|
||||||
|
|
||||||
|
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
|
||||||
|
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
|
||||||
|
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
|
||||||
|
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
|
||||||
## History
|
## History
|
||||||
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
||||||
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-02
|
created: 2026-08-02
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [decision, browser, steel, camofox, rollback]
|
tags: [decision, browser, steel, camofox, rollback]
|
||||||
sources: [session-history, live-config]
|
sources: [session-history, live-config]
|
||||||
@@ -50,7 +50,7 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
|
|||||||
> restoration was needed or wanted.
|
> restoration was needed or wanted.
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||||
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
||||||
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
|
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
|
||||||
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
|
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
|
||||||
@@ -103,6 +103,14 @@ removed**, so the `plugin_steel` toolset is no longer selectable.
|
|||||||
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
|
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
|
||||||
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
|
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
|
||||||
|
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
|
||||||
|
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
|
||||||
|
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
|
||||||
|
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
|
||||||
|
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
|
||||||
|
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
|
||||||
|
> The decision and its consequences are unchanged.
|
||||||
## Related
|
## Related
|
||||||
- [[comparisons/steel-vs-camoufox]]
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
- [[systems/browser-backend]]
|
- [[systems/browser-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-02
|
created: 2026-10-02
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, omniroute, provider, auth, incident, resolved]
|
tags: [cron, omniroute, provider, auth, incident, resolved]
|
||||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
|
||||||
@@ -130,6 +130,12 @@ done.
|
|||||||
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
|
||||||
name is absent from the provider's live catalog.
|
name is absent from the provider's live catalog.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
|
||||||
|
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
|
||||||
|
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
|
||||||
|
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
|
||||||
|
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
|
||||||
|
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
- [[incidents/2026-10-05-dead-model-pin-repin]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: CT450 Hermes Update to v0.21.5
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-04
|
created: 2026-10-04
|
||||||
updated: 2026-10-26
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, ct450, chaos, update, incident]
|
tags: [hermes, ct450, chaos, update, incident]
|
||||||
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
sources: [live-probe, pve-100.96.100.82, session-2026-10-03]
|
||||||
@@ -49,7 +49,10 @@ Two gotchas that cost time:
|
|||||||
- CT460 production untouched — separate config, skills, cron and memory state.
|
- CT460 production untouched — separate config, skills, cron and memory state.
|
||||||
|
|
||||||
## Current state (re-verified 2026-10-26)
|
## Current state (re-verified 2026-10-26)
|
||||||
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.
|
Both instances have since moved on: **CT460 and CT450 now both run `v0.22.3` (2026.9.24), local `5b147ab036d6`** — the same build. CT450 is **0 commits behind `origin/main`** (the "3 commits behind" note below is superseded), though `hermes update --check` reports stable **v0.22.5** available on both. CT450's default model is now **`smart-route`** on `custom` → `http://100.78.63.20:20128`. ~~The **CT450 Mattermost timeout remains** — CT255 is `stopped` on `pve` and its peer has been offline 36 d, so it is an infrastructure condition, not an update regression.~~ **SUPERSEDED 2026-11-18** — CT255 is `running` and its peer online; see the 11-18 state above.
|
||||||
|
|
||||||
|
## Current state (re-verified 2026-11-18)
|
||||||
|
Both instances have moved on again and are now on **different builds**: **CT460 runs `v0.23.0` (local `2bd378ed1b8d`)**, **CT450 runs `v0.23.1` (local `3b4215785882`)** — CT450 is one patch ahead and **0 commits behind `origin/main`**. The 10-26 Mattermost condition is **superseded**: **CT255 (`mattermost`) is `running` on `pve`** and its tailnet peer is `online`, so the old "CT255 stopped / peer offline 36 d" note no longer describes reality.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
|
- ~~CT450 reports `Update available: 3 commits behind`.~~ **SUPERSEDED 2026-10-26** — CT450 is now **0 commits behind `origin/main`** (it tracks the `NousResearch/hermes-agent` `main` branch; there is no `origin/master` on that remote). Re-run `hermes update` if the chaos box should track upstream exactly.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Mnemosyne Vector Recall Ran FTS-Only
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-04
|
created: 2026-10-04
|
||||||
updated: 2026-10-26
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-18
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
tags: [mnemosyne, memory, embeddings, fastembed, recall, incident]
|
||||||
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
sources: [live-probe, check_embedding_coverage.py, session-2026-10-03]
|
||||||
@@ -65,6 +65,9 @@ Both tiers are **above the 95 % working-coverage reindex threshold**, so the 6-h
|
|||||||
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
monitor job `b63ed5817212` (`last_status: ok`) has nothing to fire on. The 2026-10-04
|
||||||
figures below are superseded.
|
figures below are superseded.
|
||||||
|
|
||||||
|
## Current state (re-verified 2026-11-18)
|
||||||
|
The fix continues to hold. The 6-hourly coverage monitor `b63ed5817212` is `enabled`, `last_status: ok`, `failure_streak: 0`, last run 2026-11-18 00:04 — it has had nothing to fire on, so working coverage remains at/above the 95 % threshold. (The coverage script itself could not be re-run this session — the gateway's lifecycle guard blocks scripts touching the >1 MiB Mnemosyne DB — so the monitor's clean streak is the verification, per the monitor's own design.)
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
|
- ~~Confirm the embedding-coverage alert job (`b63ed5817212`, every 6h, Telegram topic 102) still fires when working coverage drops below 95%.~~ **CONFIRMED 2026-10-26** — the job is `enabled` with `last_status: ok`; working coverage is 96.8 %, above the threshold, so it correctly has nothing to fire on.
|
||||||
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
|
- ~~Consider asserting `fastembed` in **all** managed runtimes, not only the bootstrap-selected one, so a runtime switch cannot reintroduce this.~~ **DONE 2026-10-26** — `fastembed` 0.8.1 now imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`, `d41b16c8b6674454a4a819bd4acced1f`) as well as 0.8.0 in the legacy venv.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-06
|
created: 2026-10-06
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, model-pin, omniroute, incident, resolved]
|
tags: [cron, model-pin, omniroute, incident, resolved]
|
||||||
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
|
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
|
||||||
@@ -119,6 +119,17 @@ state rather than left on trust.
|
|||||||
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
|
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
|
||||||
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
|
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
|
||||||
|
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
|
||||||
|
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
|
||||||
|
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
|
||||||
|
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
|
||||||
|
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
|
||||||
|
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
|
||||||
|
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
|
||||||
|
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
|
||||||
|
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
|
||||||
|
> [[systems/scheduled-tasks]], not here.
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
|||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: resolved
|
||||||
created: 2026-10-09
|
created: 2026-10-09
|
||||||
updated: 2026-10-24
|
updated: 2026-11-15
|
||||||
verified_on: 2026-10-24
|
verified_on: 2026-11-15
|
||||||
review_after: 2026-11-07
|
review_after: 2026-12-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||||
@@ -18,6 +18,15 @@ sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-2026100
|
|||||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-15** (3 a.m. stocktake): all 12 of the storm's `cron_incidents` rows still read
|
||||||
|
> `resolved` — none reopened. `1b23c042e17d` (the weekly Friday job, last hold-out) is healthy —
|
||||||
|
> `last_status: ok`, last run 2026-11-13 08:11 AWST. One incident is currently open fleet-wide
|
||||||
|
> (`ea79a2_601037516abd`, Mnemosyne Health Monitor, first seen 2026-11-15 00:03) but it is the
|
||||||
|
> capture-stall detection firing → [[incidents/2026-11-14-mnemosyne-capture-stall]], unrelated to
|
||||||
|
> this storm. ⚠ Evidence window: `executions.db` retention now reaches back only to 2026-11-09, so
|
||||||
|
> the 2026-10-09 execution counts below are **no longer re-derivable from the DB** — they stand on
|
||||||
|
> the 2026-10-24 read (the incident rows themselves persist, which is what this check re-read).
|
||||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||||
|
|
||||||
## What happened
|
## What happened
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bu
|
|||||||
type: incident
|
type: incident
|
||||||
status: monitoring
|
status: monitoring
|
||||||
created: 2026-10-18
|
created: 2026-10-18
|
||||||
updated: 2026-10-27
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-27
|
verified_on: 2026-11-18
|
||||||
review_after: 2026-11-03
|
review_after: 2026-11-03
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, router, provider, infrastructure]
|
tags: [cron, router, provider, infrastructure]
|
||||||
@@ -35,6 +35,8 @@ days** (10-16 → 10-17 and 10-19) with an identical signature on an identical n
|
|||||||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||||||
> shut, the single-node exposure is not.
|
> shut, the single-node exposure is not.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-18 — 22 days clean.** Zero execution rows carry the resource-pressure string since 2026-10-27 (verified against `executions.db`). Both nodes healthy at probe time: CT600 `/api/health` → 200 with `/models` → 200 (**6,501 models**), `omni-101` `/api/health` → 200, zero unclosed `cron_incidents` rows estate-wide. Status stays **`monitoring`**: the single-node exposure and the invisible-to-`/api/health` failure mode are unchanged.
|
||||||
|
|
||||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Gateway Exited Uncleanly on 2026-10-20 — One Cron Run Orphaned (status
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-21
|
created: 2026-10-21
|
||||||
updated: 2026-10-27
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-27
|
verified_on: 2026-11-18
|
||||||
review_after: 2026-11-03
|
review_after: 2026-11-03
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
|
tags: [gateway, cron, scheduler, lifecycle, restart, silent-failure]
|
||||||
@@ -36,6 +36,8 @@ gateway exit since 2026-09-21.
|
|||||||
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
|
| 09:09:16 | log line `Starting Hermes Gateway...` (the running process reports start time 08:55:09 — the two disagree by ~14 min, so the restart instant is not pinned to the second) |
|
||||||
| ~09:10 | the orphaned execution row closes as **`unknown`** |
|
| ~09:10 | the orphaned execution row closes as **`unknown`** |
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-18 — no recurrence in the retained window.** Zero `status: unknown` executions exist in the retained `executions.db` (retention floor ~18 d, so the 10-20 row itself has aged out — its facts below stand on the 10-27 read). Retained `gateway.log*` files carry no UNCLEANLY line newer than **2026-08-31** (log rotation has since dropped the 10-20 entry the ledger once showed; per that rotation, `0` means *not in the retained window*, not *never happened*). Zero unclosed `cron_incidents` rows. Status unchanged — the silent-orphan failure mode still has no alerting of its own.
|
||||||
|
|
||||||
## Impact
|
## Impact
|
||||||
|
|
||||||
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
|
- **The 2026-10-20 Gmail Action Digest has no delivery row in `deliveries.db` and its execution carries
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||||
type: incident
|
type: incident
|
||||||
status: resolved
|
status: degraded
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-10-25
|
updated: 2026-11-15
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-11-15
|
||||||
review_after: 2026-11-08
|
review_after: 2026-12-15
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||||
@@ -14,7 +14,8 @@ sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
|||||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||||
|
|
||||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
**Status:** Resolved 2026-10-23 18:54 (patched and verified) — **but the fix was lost in the
|
||||||
|
plugin's 0.168.x restructure; `status: degraded` since 2026-11-15** (see the regression section below)
|
||||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||||
memory layer.
|
memory layer.
|
||||||
@@ -87,17 +88,39 @@ touching the file.
|
|||||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||||
is no longer abandoned by the memory layer.
|
is no longer abandoned by the memory layer.
|
||||||
|
|
||||||
|
## ⚠ Regression — the fix was lost in the plugin restructure (verified 2026-11-15)
|
||||||
|
|
||||||
|
The follow-up below warned that a plugin update would revert the patch. **It happened.** Live
|
||||||
|
state at the 2026-11-15 03:00 stocktake:
|
||||||
|
|
||||||
|
- The plugin tree is **clean at upstream `0.168.2`** (`296c94b`, committed 2026-11-12 US /
|
||||||
|
2026-11-13 AWST) — the patch is gone entirely, not just uncommitted.
|
||||||
|
- The memory layer **moved**: `dashboard/plugin_api.py` (now a doc index) →
|
||||||
|
`dashboard/api/memory.py`. There, `MEMORY_TIMEOUT_S = 10.0` is hard-coded again
|
||||||
|
(`memory.py:48`), with **no** `CONDUIT_MEMORY_TIMEOUT_S` env override, and the "abandoning the
|
||||||
|
call" behaviour is intact (`memory.py:256`). The restructure landed in `memory.py` on
|
||||||
|
**2026-10-29 AWST**, so the 60 s budget has been gone for ~2.5 weeks.
|
||||||
|
- The running dashboard (started 2026-11-10) loads the restructured code, so the exposure is live.
|
||||||
|
- One real call has since occurred — 2026-11-14 08:57 AWST, the first entry in
|
||||||
|
`conduit-calls.json → history` — with no abort lines in the logs afterwards. But that call met a
|
||||||
|
warm provider; it does not exercise the cold-start path. The page's original caveat stands.
|
||||||
|
- **Not re-applied by the stocktake**: porting the fix to the restructured code is a code change to
|
||||||
|
a git-managed upstream plugin that the next update will revert again — it needs Anthony's call
|
||||||
|
(re-port + commit locally, or upstream it).
|
||||||
|
|
||||||
|
The original incident window (2026-10-22 → 23) remains resolved; `status: degraded` records that
|
||||||
|
the *protection* is gone, not that the October fault returned.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
|
|
||||||
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
|
- ~~⚠ **The fix is still unexercised by a call** …~~ **Superseded 2026-11-15**: one call has since
|
||||||
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
|
occurred (2026-11-14 08:57 AWST) — but the fix itself was lost in the restructure (section above),
|
||||||
(`0.64.0` → `0.69.0`).
|
so "unexercised" is now moot.
|
||||||
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
|
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
|
||||||
asserted the capability did not exist **without calling the tool** →
|
asserted the capability did not exist **without calling the tool** →
|
||||||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
[[incidents/2026-10-24-conduit-call-request-refused]].
|
||||||
- **The patch is uncommitted.** A plugin update that resets
|
- ~~**The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push`
|
||||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
reverts it to the 10 s budget.~~ **Confirmed 2026-11-15** — see the regression section above.
|
||||||
(`kaishi00/hermes-conduit-notifier`).
|
|
||||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Conduit Call Requests Refused by the Agent
|
|||||||
type: incident
|
type: incident
|
||||||
status: open
|
status: open
|
||||||
created: 2026-10-25
|
created: 2026-10-25
|
||||||
updated: 2026-10-26
|
updated: 2026-11-19
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-11-19
|
||||||
review_after: 2026-11-08
|
review_after: 2026-11-08
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
|
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
|
||||||
@@ -93,3 +93,11 @@ into [[incidents/2026-10-22-conduit-memory-start-timeout]].
|
|||||||
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
- The plugin **is** working for cards: `conduit_show` ×1 on 2026-10-25 14:37, delivered live.
|
||||||
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
- Plugin advanced to **`0.69.2`** and the dashboard gateway restarted at **18:42:33**, so upstream commits are live — see [[systems/conduit]].
|
||||||
- **Status stays open**: the request path is untested, not confirmed good.
|
- **Status stays open**: the request path is untested, not confirmed good.
|
||||||
|
|
||||||
|
## Status update — 2026-11-19 (stocktake re-probe)
|
||||||
|
|
||||||
|
- `~/.hermes/conduit-calls.json` → `history` is **still empty (0 entries)** as of 2026-11-19 03:0x
|
||||||
|
AWST — the call path has now gone unused for ~26 days since the fault. The "verify a real call
|
||||||
|
end-to-end" open item remains outstanding.
|
||||||
|
- No new refusal episodes have been recorded against this page. **Status stays open** on the same
|
||||||
|
basis as before: untested, not confirmed good.
|
||||||
|
|||||||
@@ -1,13 +1,13 @@
|
|||||||
---
|
---
|
||||||
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
title: Gmail Action Digest — 1 h 55 m Runaway and Six-Fire Refire Loop
|
||||||
type: incident
|
type: incident
|
||||||
status: open
|
status: resolved
|
||||||
created: 2026-10-26
|
created: 2026-10-26
|
||||||
updated: 2026-10-26
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-18
|
||||||
review_after: 2026-11-09
|
review_after: 2026-11-09
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [cron, automation, hermes, incident, unresolved]
|
tags: [cron, automation, hermes, incident]
|
||||||
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
sources: [live-sqlite-cron-executions, live-sqlite-sessions, gateway-log, jobs-json]
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -55,6 +55,9 @@ scheduler admitted an unrelated session is unexplained and is the first thing to
|
|||||||
execution row.
|
execution row.
|
||||||
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
- Does `delivery_queued` clear on the 2026-10-26 08:30 slot? Check after that fire.
|
||||||
|
|
||||||
|
## Resolution (verified 2026-11-18)
|
||||||
|
**Resolved — no recurrence in 23 days.** The `delivery_queued` state cleared (the open question above is answered: it did **not** persist past the 2026-10-26 08:30 slot). Job `08eb86f02d37` now reads `enabled`, `last_status: ok`, `last_error: null`, `failure_streak: 0`, last run 2026-11-17 08:33. In the retained `executions.db` window (2026-11-12 → 11-17; DB retention floor ~18 d means earlier slots have aged out) it ran **exactly once per day, `completed` every time, zero failed rows** — no refires since the 10-25 loop. Root cause of the 10-25 session-adoption remains hypothesised, not confirmed; the wall-clock-budget follow-up was never implemented, so the *exposure* (a runaway can recur silently) is unchanged even though the incident itself has not recurred.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
- [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: resolved
|
||||||
created: 2026-10-27
|
created: 2026-10-27
|
||||||
updated: 2026-10-28
|
updated: 2026-11-19
|
||||||
verified_on: 2026-10-28
|
verified_on: 2026-11-19
|
||||||
review_after: 2026-11-03
|
review_after: 2026-12-19
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
|
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe]
|
||||||
@@ -111,3 +111,15 @@ disappears silently. What it did not do is *finish the work* or *avoid the inter
|
|||||||
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
- The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row.
|
||||||
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
- Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline
|
||||||
(`timed_out=True` at 1.35 s of a 30 s budget).
|
(`timed_out=True` at 1.35 s of a 30 s budget).
|
||||||
|
|
||||||
|
## Update — 2026-11-19 (stocktake re-probe)
|
||||||
|
|
||||||
|
- **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3`
|
||||||
|
`resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13**
|
||||||
|
(live probe of `cron_incidents` in `~/.hermes/cron/executions.db`).
|
||||||
|
- The `lost its durable fire claim ownership` error string has **no rows in the retained
|
||||||
|
`executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention
|
||||||
|
floor; the 10-28 update already confirmed the 10-27 window clean).
|
||||||
|
- **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups
|
||||||
|
1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics)
|
||||||
|
remain open design questions and were **not** actioned.
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-11-09
|
created: 2026-11-09
|
||||||
updated: 2026-11-14
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-14
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
|
||||||
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
|
||||||
@@ -128,6 +128,33 @@ insufficient at current load**, and this page is no longer `resolved`.
|
|||||||
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
fault.** The remaining levers are unchanged and now urgent — **`state.db` checkpoint/retention** and
|
||||||
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
reducing host load (see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
|
||||||
|
|
||||||
|
## Recurrence 2 (2026-11-15) — eleven timeouts in two clusters, 10 refusals
|
||||||
|
|
||||||
|
Recorded by the **2026-11-17 nightly audit** while backing up the 11-15 window: the **2026-11-15 audit
|
||||||
|
captured none of this**. Exact `errors.log` counts for 11-15:
|
||||||
|
|
||||||
|
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping` —
|
||||||
|
**5** in 01:07:17 → 01:25:43 and **6** in 12:18:15 → 12:48:40 (a two-cluster day, unlike 11-14's one).
|
||||||
|
- **10** tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||||
|
— hour 01: `read_file` ×2 (134.17 s and 0.43 s), `execute_code` ×4, `patch` ×1, `read_file` ×1
|
||||||
|
(121.03 s); hour 12: `execute_code` ×1. Call-level blocks reach **120.63 s**, and the affected sessions
|
||||||
|
are `local-989b3642…` (01:xx) and `local-b6a1490f…` (12:xx).
|
||||||
|
- 11-14, for comparison: **1** timeout and **4** refusals, plus **4** plugin load failures.
|
||||||
|
- **11-16 and 11-17: 0 timeouts, 0 refusals, 0 load failures.** The episode did not carry over.
|
||||||
|
|
||||||
|
The loader half of the same 11-15 storm — **52** failed plugin loads, **519** discarded registrations
|
||||||
|
(`conduit_push` 474 of them) and **2** `pre_llm_call` timeouts — is now on its own page,
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]]. Its 12:18–12:49 cluster is the useful detail here:
|
||||||
|
**a hook-timeout cluster can occur with no accompanying loader failure**, so the two share a cause class
|
||||||
|
rather than a mechanism.
|
||||||
|
|
||||||
|
**Effect on the trigger hypothesis recorded above.** Amplitude is not a simple function of load average:
|
||||||
|
the 11-15 episode ran at a load the audit cannot measure retroactively, while the 11-17 read
|
||||||
|
(**6.19 / 6.64 / 7.48**) sits *below* the 11-14 figure of 12.08 with **no** recurrence — and a *larger*
|
||||||
|
`state.db` (2.09 GB, WAL 40 MB, `/` at 88 %). Keep a **load-average threshold out of this page's claim**;
|
||||||
|
keep **`state.db` size, disk headroom and the gateway's own timing budgets** in it.
|
||||||
|
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
||||||
- [[systems/conduit]]
|
- [[systems/conduit]]
|
||||||
|
|||||||
@@ -1,17 +1,21 @@
|
|||||||
---
|
---
|
||||||
title: Mnemosyne capture stalled for ~15 hours (2026-11-14 09:33 AWST onward) — health monitor escalated 00:03 AWST 2026-11-15
|
title: Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-11-15
|
created: 2026-11-15
|
||||||
updated: 2026-11-15
|
updated: 2026-11-20
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-20
|
||||||
review_after: 2026-11-17
|
review_after: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident]
|
tags: [mnemosyne, memory, capture, cron, health-monitor, desktop, pulse, incident, signature-fragmentation, freshness-gate]
|
||||||
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db]
|
sources: [live-mnemosyne-db, live-cron-executions-db, live-cron-incidents, live-hermes-agent-log, live-mnemosyne-health-monitor-log, live-config-yaml, live-state-db, live-mnemosyne-freshness-gate-script]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Mnemosyne capture stalled for ~15 hours (from 2026-11-14 09:33 AWST)
|
# Mnemosyne capture stalled 105 h 52 m (2026-11-14 09:33 → 2026-11-18 19:25 AWST), then quiet again — four unclosed alerts
|
||||||
|
|
||||||
|
> ⚠ **2026-11-20 re-read — the freeze ended by itself at 2026-11-18 19:25 AWST (105 h 52 m total), and every
|
||||||
|
> interactive session inside the window was captured.** The "capture is dead" reading below is a *quiet estate*,
|
||||||
|
> not a broken hook. Read the newest re-read section before the earlier readings.
|
||||||
|
|
||||||
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
**Open.** Nothing has been written to Mnemosyne's capture tables since **2026-11-14 ~09:33 AWST**,
|
||||||
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
and on **2026-11-15 00:03:35** the *Mnemosyne Health Monitor* (`ea79a2107e36`) **failed** on it
|
||||||
@@ -21,6 +25,181 @@ Found by the **2026-11-14 nightly audit**; not caused by anything in that audit'
|
|||||||
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
the previous morning and simply had not been noticed, because the monitor's own earlier reading of
|
||||||
the same condition was classified informational — see below).
|
the same condition was classified informational — see below).
|
||||||
|
|
||||||
|
## Re-read 2026-11-16 00:18 AWST — still open, stall 38 h 46 m, and now **three** unclosed alerts
|
||||||
|
|
||||||
|
Six hours later, at the next nightly audit, nothing has moved. The freeze is **total**, the monitor has
|
||||||
|
fired twice more, and the scheduler has minted further unclosed rows for the same fault.
|
||||||
|
|
||||||
|
- **The counts are frozen, not merely low.** `working_memory` **1130** / `episodic_memory` **1249** —
|
||||||
|
**the same two numbers as every earlier read of the stall** (2026-11-15 ~00:4x and 12:03) — so this is
|
||||||
|
not a quiet stretch or a degraded trickle. Zero writes in **38 h 46 m**.
|
||||||
|
- **The whole write path stopped, not just turn capture.** Newest rows remain the three
|
||||||
|
`[MODEL_REFRESH_PROPOSAL]` maintenance writes at **`09:32:58` / `09:32:59` AWST**; the newest
|
||||||
|
*user-turn* capture is **`2026-11-14 09:04:19 AWST`**. Maintenance outlived the last turn by ~28 min,
|
||||||
|
then both stopped inside the same minute.
|
||||||
|
- **`cron_incidents` 1 → 3 unclosed** (138 → **141** rows total). All three are `ea79a2107e36`, all
|
||||||
|
`failure_type: auth`, all `state: alerted`, and **each has `first_seen_at == last_seen_at`** — a single
|
||||||
|
observation, no `last_seen_at` accumulation, no escalation:
|
||||||
|
|
||||||
|
| Incident id | `first_seen_at` | `alerted_at` |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `ea79a2_601037516abd` | `2026-11-15T00:03:35.815373+08:00` | `2026-11-15T00:03:44.559501+08:00` |
|
||||||
|
| `ea79a2_5f817708701e` | `2026-11-15T12:03:14.928500+08:00` | `2026-11-15T12:03:20.245081+08:00` |
|
||||||
|
| `ea79a2_75303493d484` | `2026-11-16T00:03:15.418001+08:00` | `2026-11-16T00:03:22.365399+08:00` |
|
||||||
|
|
||||||
|
- ⚠ **New mechanism, evidenced: the incident signature embeds the elapsed-hours float.** The three
|
||||||
|
signatures differ only by `4.64157453` / `4.634034074444445` / `4.643481075555555` — a value **never
|
||||||
|
equal twice** — so the recurrence never matches the prior row. One durable fault therefore **mints a
|
||||||
|
fresh unclosed row every slot**, and the estate's unclosed count grows ~1 per 12 h while no single row
|
||||||
|
ever escalates. This job alone now has **15** `cron_incidents` rows (12 older `unknown`, these 3
|
||||||
|
`auth`). **Proposed fix — NOT applied** (it changes the monitor's incident path, so it is Anthony's
|
||||||
|
call): normalise the value out of the signature — round it, or omit it — so the same condition
|
||||||
|
**re-opens the existing row** instead of fragmenting.
|
||||||
|
|
||||||
|
### The monitor's ≈4.64 h figure is NOT the stall — verified against the gate's own source
|
||||||
|
|
||||||
|
Every alert in this sequence prints a near-identical elapsed time while the stall grew from 14 h to
|
||||||
|
38.7 h. That number is not a measurement of the stall. `mnemosyne_freshness_gate.py`'s
|
||||||
|
`newest_user_message()` selects the newest `messages` row with `role='user'` whose session
|
||||||
|
`source NOT IN ('cron','subagent')` (the `NON_CONVERSATIONAL` set, line 52) and returns that row's
|
||||||
|
`source`; the alert prints **`(now − that message) / 3600` h**. So the figure is **the age of the newest
|
||||||
|
*conversational* message — not the age of the newest capture.**
|
||||||
|
|
||||||
|
The newest conversational messages on this box are **Pulse** turns recorded under `source: desktop`
|
||||||
|
(`Pulse delivery check pXXXX-1 …`, `Pulse morning brief …`), on a **~6 h cadence**. Each alert's
|
||||||
|
≈4.64 h lands on the then-newest Pulse turn to within about a minute:
|
||||||
|
|
||||||
|
| Alert | Elapsed printed | Now ≈ | Newest conversational message it measured |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 2026-11-15 00:03:35 | `4.64157 h` | 00:00:54 | **2026-11-14 19:22:30** (desktop / Pulse) |
|
||||||
|
| 2026-11-15 12:03:14 | `4.63403 h` | 12:01:25 | **2026-11-15 07:23:25** (desktop / Pulse) |
|
||||||
|
| 2026-11-16 00:03:15 | `4.64348 h` | 00:01:33 | **2026-11-15 19:22:59** (desktop / Pulse) |
|
||||||
|
|
||||||
|
**Why they all read ≈4.64:** the `0 */12` slots happen to fire ~4.6 h after the most recent Pulse turn.
|
||||||
|
The near-constancy is a **cadence coincidence** and must **not** be read as "the stall is only ~4.6 h" —
|
||||||
|
the stall is **38 h 46 m**. Same trap class as the naive-UTC note below: a number shaped like a duration
|
||||||
|
that is not one.
|
||||||
|
|
||||||
|
**Positive proof the capture path is dead rather than merely idle:** the identical class of message *was*
|
||||||
|
captured before the stall — the 2026-11-14 00:09:12 run logged
|
||||||
|
`last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`. Four 2026-11-15 Pulse turns
|
||||||
|
(`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have **no capture row at all**.
|
||||||
|
|
||||||
|
### The consolidator is alive and every cycle is a deliberate no-op
|
||||||
|
|
||||||
|
Latest `consolidation_log` row **4696** at `2026-11-15T16:14:14.934306+00:00` (= **2026-11-16 00:14:14
|
||||||
|
AWST**), `items_consolidated: 0`, `session_id: auto-consolidate-maintenance`; cadence ~31 min
|
||||||
|
(`15:12:13` → `15:43:08` → `16:14:14` UTC). **New evidence:** the row's `summary_preview` reads
|
||||||
|
**`No-op consolidation cycle — timestamp updated for watchdog`** — the cycle **touches its own timestamp
|
||||||
|
to satisfy an age test**, so a zero-item streak can run indefinitely while every age-based gate reads
|
||||||
|
"fresh". The streak is now **~74 consecutive zero-item cycles**, opening
|
||||||
|
`2026-11-14T02:04:42+00:00` (10:04:42 AWST). **A zero-item-streak threshold — not an age threshold — is
|
||||||
|
what would have caught this at 10:04 on 11-14.**
|
||||||
|
|
||||||
|
### Adjacent fault in the same runtime — the watchdog lost a slot (2026-11-15 11:31)
|
||||||
|
|
||||||
|
`96eb95ce12f2` (**Mnemosyne Watchdog**, every 15 min) **failed** on its `11:31:20.699` slot,
|
||||||
|
`finished_at 2026-11-15T11:49:10.534610` — **17 m 49.8 s**, exit 1:
|
||||||
|
|
||||||
|
```
|
||||||
|
runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
A **600 s timeout inside a 15-minute slot** overran two slots: the `11:45` slot was **skipped** (no row
|
||||||
|
at all) and the **next slot, `12:01:25.120952`, completed** (`→ 12:02:02.657231`), so it self-recovered
|
||||||
|
with one slot lost. This is a **new signature** for this job — the August and October occurrences had the
|
||||||
|
guard quoting *another* job's error text. Across 11-15 → 11-16 it is the only `failed` row of its 97.
|
||||||
|
Alert delivered (`deliveries.db` `11:49:06.052341`, `for_failure: 1`, `delivered`). **Adjacency, not
|
||||||
|
proof of common cause** — but the guard and the capture path share the same runtime, so a guard that
|
||||||
|
times out at 600 s belongs next to this stall rather than dismissed.
|
||||||
|
|
||||||
|
|
||||||
|
## Re-read 2026-11-17 00:27 AWST — stall **62 h 54 m**, **five** unclosed alerts, zero-item streak **120**
|
||||||
|
|
||||||
|
Third nightly read, sixth week of the same numbers changing only their counters. **The freeze is still
|
||||||
|
total.**
|
||||||
|
|
||||||
|
- **Counts identical for the fifth consecutive read:** `working_memory` **1130** / `episodic_memory`
|
||||||
|
**1249**; newest rows still `2026-11-14 09:32:59` AWST (working) and `09:33:34` AWST (episodic) —
|
||||||
|
**62 h 54 m** with **zero** writes.
|
||||||
|
- **`cron_incidents` 3 → 5 unclosed** (141 → **143** rows total). Both new rows are `ea79a2107e36`,
|
||||||
|
`failure_type: auth`, `state: alerted`, and again `first_seen_at == last_seen_at`:
|
||||||
|
|
||||||
|
| Incident id | `first_seen_at` | `alerted_at` |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `ea79a2_915d004a108a` | `2026-11-16T12:02:06.679532+08:00` | `2026-11-16T12:02:12.472265+08:00` |
|
||||||
|
| `ea79a2_0fab8d43db75` | `2026-11-17T00:03:37.624640+08:00` | `2026-11-17T00:03:46.151392+08:00` |
|
||||||
|
|
||||||
|
The signature-fragmentation mechanism below is holding exactly as described: **one fresh row per
|
||||||
|
firing, no escalation**, so the estate's unclosed count now grows **~2 per day** with nothing accumulating
|
||||||
|
on any single row.
|
||||||
|
- **The monitor is on a 5-slot failure streak** (`jobs.json`: `ea79a2107e36`, `last_status: error`,
|
||||||
|
`failure_streak: 5`). 11-16's ledger holds **2 failed rows of 175, both this job**
|
||||||
|
(00:01:33 → 00:03:22, 12:01:18 → 12:02:12); 11-17 so far 1 failed of 7 (00:01:39 → 00:03:46).
|
||||||
|
Every failure is `exit 1` with `STALE MEMORY PIPELINE: capture missed: us…` — the same gate.
|
||||||
|
- **The consolidator is alive and deliberately doing nothing.** `consolidation_log` latest row **4742**
|
||||||
|
at `2026-11-16T16:00:41.922076+00:00` (= 00:00:41 AWST), `items_consolidated: 0`, summary *"No-op
|
||||||
|
consolidation cycle — timestamp updated for watchdog"*. Last non-zero row: **4622** at
|
||||||
|
`2026-11-14T09:33:34` → **120 consecutive zero-item cycles** (was 74 at the 11-16 read). The cycles
|
||||||
|
still fire ~every 31 min; there is simply **nothing new to consolidate**, which is the memory-side
|
||||||
|
reason the stall cannot self-heal.
|
||||||
|
- **Still no cause, still no fix, still `status: active`.** The 11-16 read's two conclusions stand: the
|
||||||
|
failure is **total**, not a trickle, and it affects **all** write paths, not just turn capture.
|
||||||
|
`review_after` 2026-11-18 — this page is re-read at the next audit, and the monitor keeps alerting.
|
||||||
|
|
||||||
|
|
||||||
|
## Re-read 2026-11-20 00:5x AWST — the freeze ended by itself after **105 h 52 m**, and the "capture is dead" reading was wrong
|
||||||
|
|
||||||
|
**The capture freeze is over. The alarm is not.** Live reads of `mnemosyne.db` at 2026-11-20 00:0x–00:5x AWST:
|
||||||
|
|
||||||
|
| counter | 2026-11-17 00:27 | 2026-11-20 00:5x |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `working_memory` | 1130 | **1131** |
|
||||||
|
| `episodic_memory` | 1249 | **1250** |
|
||||||
|
| newest write | `2026-11-14 09:32:59` AWST | **`2026-11-18 19:25:03` AWST** (stored UTC `11:25:03`) |
|
||||||
|
|
||||||
|
The break coincides with nothing: no config edit, no restart and no repair is recorded against the capture path
|
||||||
|
in the window — the only nearby state change is the 11-18 19:18 desktop session itself. The single new
|
||||||
|
`working_memory` row carries `session_id: hermes_20261118_191808_5c9e08` and its content begins
|
||||||
|
`[USER] You are helping me migrate contex…`, which identifies the captured turn (the `desktop` session
|
||||||
|
**`20261118_191808_5c9e08`**, *"Helping migrate context AI assistant"*, 9 messages, started 19:18 AWST).
|
||||||
|
**Stall duration: 105 h 52 m** (2026-11-14 09:33 → 2026-11-18 19:25 AWST).
|
||||||
|
|
||||||
|
⚠ **The correction that matters — the window contained almost no conversation to capture.** `state.db` lists
|
||||||
|
**three** non-cron sessions since 2026-11-14 00:00 AWST: two `desktop` sessions on **11-14** (**08:35**, 40
|
||||||
|
messages; **08:52**, 41 messages — both *before* the 09:33 boundary), then **nothing** until the **11-18 19:18**
|
||||||
|
session, whose turn is the row that ends the freeze. Every interactive session in the window was captured, so
|
||||||
|
the earlier figures on this page (`0 writes in 38 h 46 m`, `62 h 54 m`) are real but measure **a quiet estate**,
|
||||||
|
not a broken hook: `mnemosyne_freshness_gate.py`'s `newest_user_message()` reports the age of the newest message
|
||||||
|
whose session source is not `cron`/`subagent`, so **silence and failure read identically**. That ambiguity — not
|
||||||
|
a broken capture path — is the durable defect on this page.
|
||||||
|
|
||||||
|
**The monitor is still failing on that same metric, after the break.** `ea79a2107e36` slots since 11-18:
|
||||||
|
`11-18 00:01:28` failed · `11-18 12:01:13` failed · **`11-19 00:01:34` `completed`** (the only clean slot since
|
||||||
|
11-17) · `11-19 12:01:17` failed · `11-20 00:01:22` failed — every failure `Script exited with code 1 / 🚨
|
||||||
|
Mnemosyne Health Monitor — exception(s): STALE MEMORY PIPELINE: capture …`, i.e. **the same signature firing on
|
||||||
|
a healthy pipeline** (`failure_streak: 2`; incidents `ea79a2_f2be2db3bfce` at 2026-11-19 12:01:58 and
|
||||||
|
`ea79a2_aae4d914f3c9` at 2026-11-20 00:03:36, both `alerted`, both `failure_type: auth` — the scheduler's
|
||||||
|
classifier, not a credential fault). The `11-19 00:01` pass and the `12:01` failure bracket the threshold: the
|
||||||
|
metric is *age of the newest captured conversational turn*, which only grows while the estate is silent.
|
||||||
|
|
||||||
|
**The other half of the original finding is unchanged, and larger.** The consolidator still consolidates zero
|
||||||
|
items: newest `consolidation_log` row **`2026-11-19T16:20:14.587267+00:00`** (= 2026-11-20 00:20 AWST, so it is
|
||||||
|
running on its ~31-minute cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle
|
||||||
|
— timestamp updated for watchdog"`. **199 of the last 200 cycles are zero-item** (was **120** at the 11-17 read
|
||||||
|
— consistent with ~31 min cadence over the three days). The zero-item-streak threshold remains **proposed, not
|
||||||
|
applied**.
|
||||||
|
|
||||||
|
**Unclosed incidents: 5 → 4** (one of the rows present at the 11-17 read has since closed). Current four, none
|
||||||
|
closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, `alerted_at: None` — the wiki audit's
|
||||||
|
silent skip, see [[incidents/2026-11-18-wiki-audit-credit-outage]]), `96eb95_98ade422d195` (*script*,
|
||||||
|
2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16 — see
|
||||||
|
[[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]), `ea79a2_f2be2db3bfce` (2026-11-19
|
||||||
|
12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36).
|
||||||
|
|
||||||
|
**Standing open question, restated:** not *"why did capture stop"* but **"why does the gate treat conversational
|
||||||
|
silence as a fault?"** Until that is fixed, this page will keep re-opening on quiet days.
|
||||||
|
|
||||||
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
## Symptom (live reads, 2026-11-15 ~00:3x–00:4x AWST)
|
||||||
|
|
||||||
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
- Newest `working_memory.created_at`: **`2026-11-14 01:32:59`** — stored **UTC**, i.e.
|
||||||
@@ -46,28 +225,68 @@ has no working_memory row (last capture 2026-11-14 01:32:59)
|
|||||||
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
|
||||||
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
|
||||||
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
|
||||||
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
|
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
|
||||||
the page records no auth error text. **Observation, not verified diagnosis.**
|
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
|
||||||
|
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
|
||||||
|
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
|
||||||
|
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
|
||||||
|
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
|
||||||
|
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
|
||||||
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
|
||||||
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
|
||||||
log line below.
|
log line below.
|
||||||
|
|
||||||
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
|
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
|
||||||
|
|
||||||
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
|
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
|
||||||
same hole and calling it benign:
|
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
|
||||||
|
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
|
||||||
|
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
|
||||||
|
|
||||||
```
|
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
|
||||||
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
|
|
||||||
```
|
|
||||||
|
|
||||||
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
|
| line | content |
|
||||||
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
|
| --- | --- |
|
||||||
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
|
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
|
||||||
**So the fault had ~12 hours of lead time that produced no alarm.**
|
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
|
||||||
|
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
|
||||||
|
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
|
||||||
|
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
|
||||||
|
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
|
||||||
|
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
|
||||||
|
|
||||||
|
Two further corrections to the earlier revision:
|
||||||
|
|
||||||
|
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
|
||||||
|
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
|
||||||
|
sentence is withdrawn — 18:03 belongs to some other job.
|
||||||
|
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
|
||||||
|
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
|
||||||
|
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
|
||||||
|
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
|
||||||
|
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
|
||||||
|
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
|
||||||
|
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
|
||||||
|
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
|
||||||
|
(`-589.5 min` = 09:32:59 − 19:22).
|
||||||
|
|
||||||
|
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
|
||||||
|
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
|
||||||
|
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
|
||||||
|
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
|
||||||
|
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
|
||||||
|
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
|
||||||
|
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
|
||||||
|
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
|
||||||
|
the loudest signal available and the only one nothing watches.**
|
||||||
|
|
||||||
## Impact
|
## Impact
|
||||||
|
|
||||||
|
> **Magnitude superseded 2026-11-16 00:18 AWST: 38 h 46 m uncaptured** (was ~15 h), with **two more
|
||||||
|
> unclosed alerts** and four 2026-11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`)
|
||||||
|
> added to the uncaptured set. The ~15 h figures below stand as the 2026-11-15 readings.
|
||||||
|
|
||||||
|
|
||||||
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
- ~15 hours of context is **not in memory**: the desktop/Bot Chat session that ran to **19:22 AWST**
|
||||||
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
(the last user message the monitor could see), all Telegram traffic, and every cron session in the
|
||||||
window. Recall over that period will answer from nothing.
|
window. Recall over that period will answer from nothing.
|
||||||
@@ -98,7 +317,12 @@ The previously recorded capture-killer on this host is a **stringified list/dict
|
|||||||
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
**`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — i.e. **3 s after** the
|
||||||
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
incident row was raised and **5 s before** its `alerted_at` (00:03:44.5). The notification path worked;
|
||||||
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
the **detection** path did not (see the ℹ️-at-12:00 bullet on [[systems/scheduled-tasks]]).
|
||||||
- Whether capture resumed after this audit's read (the monitor's next slot is the natural check).
|
- ✅ **Answered 2026-11-16 00:18 — capture did NOT resume.** Two further monitor slots failed on the
|
||||||
|
same condition (`12:03:14` and `2026-11-16 00:03:15`) and the row counts are unchanged (`working`
|
||||||
|
1130 / `episodic` 1249). Closed as *no*; the underlying cause remains open.
|
||||||
|
- **The estate's only automated alarm is understating this incident.** The alert's ≈4.64 h is a cadence
|
||||||
|
artefact, not a stall duration — anyone reading the alert text alone would conclude the pipeline is
|
||||||
|
~4.6 h behind when it is 38 h 46 m. A stall-duration figure (newest capture vs now) would fix it.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
---
|
||||||
|
title: Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||||
|
type: incident
|
||||||
|
status: unresolved
|
||||||
|
created: 2026-11-17
|
||||||
|
updated: 2026-11-17
|
||||||
|
verified_on: 2026-11-17
|
||||||
|
review_after: 2026-11-24
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, plugin, conduit, timeout, infrastructure, incident, load]
|
||||||
|
sources: [live-errors-log, live-config-yaml, live-gateway-log, live-cron-jobs-json]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Plugin load-timeout storm — 52 failed loads and 519 dropped registrations (2026-11-15)
|
||||||
|
|
||||||
|
**Not observed on 2026-11-16 or 2026-11-17 — 0 lines of either signature on both days.** No fix was
|
||||||
|
applied; the storm stopped as abruptly as it started, and the loader budget that produced it is
|
||||||
|
unchanged, so recurrence is possible. Found by the **2026-11-17 nightly audit**, which was filling the
|
||||||
|
11-15 window's gaps; the **2026-11-15 audit recorded none of this**. Every count below is an exact
|
||||||
|
`errors.log` grep, not an estimate.
|
||||||
|
|
||||||
|
## What the log records (2026-11-15)
|
||||||
|
|
||||||
|
- **52** × `Failed to load plugin '<name>': load timed out after 10s (import + register() never returned)`
|
||||||
|
- **519** × `Plugin '<name>' called register_{tool,hook,skill,platform}() after its load timed out; ignored`
|
||||||
|
- **11** × `Hook 'pre_tool_call' callback _pre_tool_call timed out after 120s — skipping`
|
||||||
|
- **2** × `Hook 'pre_llm_call' callback _pre_llm_call timed out after 120s — skipping` (06:05:33, 08:41:41)
|
||||||
|
- **10** × tool calls returned `{"error": "pre_tool_call plugin callback timed out or is still running"}`
|
||||||
|
|
||||||
|
Across the log's retained window (`errors.log` starts **2026-11-09 13:18**; `.1` / `.2` rotations hold
|
||||||
|
older lines):
|
||||||
|
|
||||||
|
| Day | Plugin load failures | Registrations dropped | `pre_tool_call` timeouts | `pre_llm_call` timeouts | Tool refusals |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 2026-11-12 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-13 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-14 | 4 | 0 | 1 | 1 | 4 |
|
||||||
|
| **2026-11-15** | **52** | **519** | **11** | **2** | **10** |
|
||||||
|
| 2026-11-16 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
| 2026-11-17 | 0 | 0 | 0 | 0 | 0 |
|
||||||
|
|
||||||
|
### By plugin (2026-11-15)
|
||||||
|
|
||||||
|
| Plugin | Failed loads | Registrations dropped |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `hermes-hq-push` | 26 | 0 |
|
||||||
|
| `conduit_push` | 9 | **474** |
|
||||||
|
| `homeassistant` | 7 | 34 |
|
||||||
|
| `browser-browser-use` | 3 | 3 |
|
||||||
|
| `basic` | 2 | 2 |
|
||||||
|
| `cadu-secrets-vault` | 1 | 4 |
|
||||||
|
| `telegram-platform` | 1 | 1 |
|
||||||
|
| `nous` | 1 | 0 |
|
||||||
|
| `drain` | 1 | 0 |
|
||||||
|
| `browser-browserbase` | 1 | 1 |
|
||||||
|
| **Total** | **52** | **519** |
|
||||||
|
|
||||||
|
## The dropped registrations are the more serious half
|
||||||
|
|
||||||
|
A failed load is a warning line. The `register_*()` lines that follow are **function loss**: the loader
|
||||||
|
gives each plugin a **10 s** budget for `import + register()`, and when that budget expires the plugin's
|
||||||
|
late registrations are **discarded, not honoured**. `conduit_push` alone lost **474** registrations across
|
||||||
|
**eight** load events (01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47) — i.e. for those
|
||||||
|
processes the Conduit tools, skills and hooks were **not registered at all**, silently. See
|
||||||
|
[[systems/conduit]].
|
||||||
|
|
||||||
|
## Clustering
|
||||||
|
|
||||||
|
These arrive in bursts, not as a trickle. 18 distinct registration-drop groups on 11-15; load failures
|
||||||
|
spread over hours 01, 04–11 and 13:
|
||||||
|
|
||||||
|
- **01:05–01:32** — `hermes-hq-push` (01:05, 01:17, 01:32) and `homeassistant` (01:17) fail to load,
|
||||||
|
`conduit_push` fails at 01:32:39 and drops ~52 registrations. Inside the same 20 minutes
|
||||||
|
`pre_tool_call` times out **5×** (01:07:17, 01:16:11, 01:19:36, 01:22:39, 01:25:43) and **9** tool calls
|
||||||
|
are refused in the same session (`local-989b3642…`).
|
||||||
|
- **04:46–04:47** — the day's largest burst: `browser-browser-use`, `conduit_push` ×2, `hermes-hq-push` ×2
|
||||||
|
and `homeassistant` all fail within ~55 s; **118** registrations discarded.
|
||||||
|
- **06:31–06:48, 07:06, 08:16, 08:46, 10:47** — repeated `conduit_push`/`hermes-hq-push` failures, each
|
||||||
|
dropping ~52–56 registrations.
|
||||||
|
- **12:18–12:49** — a **second** `pre_tool_call` cluster (6 timeouts + 1 refusal) with **no load failures**
|
||||||
|
in hours 11–12 bar one at 11:49 — so the hook half of this page's family can fire without a loader
|
||||||
|
failure alongside it.
|
||||||
|
|
||||||
|
## What this is — and is not
|
||||||
|
|
||||||
|
- **Not a misconfiguration.** `plugins.hook_callback_timeout: 120` is set in `config.yaml` and **no**
|
||||||
|
plugin-load-timeout key exists there — the 10 s loader budget is a **code default**. The failures are a
|
||||||
|
**saturation symptom** on this host, not a wrong setting.
|
||||||
|
- **Same window as the hook cascade.** [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||||
|
recurred on 11-15 with the 11 `pre_tool_call` timeouts and 10 refusals counted above; the runtime deps
|
||||||
|
guard also blew its **600 s** budget that day (see
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]). Several of the estate's own timing budgets were
|
||||||
|
exceeded in the same 24 h.
|
||||||
|
- **Adjacency, not proof.** Nothing ties the three to a single cause. What they share is a host under
|
||||||
|
sustained IO/scheduler pressure while `state.db` had grown past **2 GB** with a **40 MB** WAL (`/` at
|
||||||
|
88 %). But load average at the 2026-11-17 audit read **6.19 / 6.64 / 7.48** — *below* the 11-14 reading
|
||||||
|
of 12.08 — with **no** recurrence of this class on 11-16/11-17, so **load average alone does not
|
||||||
|
predict it**.
|
||||||
|
- **Not host restarts.** `gateway.log` has **0** startup/shutdown markers on 11-15, and `uptime` at the
|
||||||
|
audit was 6 d 2 h (host last up **2026-11-10 ~22:04**) — so the ~18 load events are **in-process plugin
|
||||||
|
(re)loads**. Plugin registration output in `agent.log` appears at agent-process start (e.g. the
|
||||||
|
2026-11-16 00:00:00 block), which is why a job starting during saturation can run with plugins missing.
|
||||||
|
|
||||||
|
## Open question
|
||||||
|
|
||||||
|
Why did `import + register()` exceed 10 s for whole plugins, and does the loader fail **closed or open**
|
||||||
|
for the caller? Today it fails **open** — the job continues, silently without those tools. A check that
|
||||||
|
asserts **plugin registration** rather than only job exit status would surface this class; the same idea
|
||||||
|
applies to the hook callback budget. **Proposed, not applied** (it changes Hermes runtime behaviour, so
|
||||||
|
it is Anthony's call).
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — the companion hook-callback cascade
|
||||||
|
- [[systems/conduit]] — what the 474 dropped `conduit_push` registrations cost
|
||||||
|
- [[systems/hermes-agent]] — plugin loading, config and timeouts
|
||||||
|
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — same-window runtime pressure
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
---
|
||||||
|
title: Mnemosyne Watchdog red for 115 consecutive 15-minute slots — 59 unresolved conflicts against a threshold of 50 (from 2026-11-18 20:01 AWST)
|
||||||
|
type: incident
|
||||||
|
status: active
|
||||||
|
created: 2026-11-20
|
||||||
|
updated: 2026-11-20
|
||||||
|
verified_on: 2026-11-20
|
||||||
|
review_after: 2026-11-21
|
||||||
|
confidence: high
|
||||||
|
tags: [mnemosyne, memory, conflicts, watchdog, cron, alert-fatigue, incident]
|
||||||
|
sources: [live-cron-executions-db, live-cron-incidents, live-mnemosyne-db, live-deliveries-db, live-jobs-json]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mnemosyne Watchdog red for 115 consecutive slots (from 2026-11-18 20:01 AWST)
|
||||||
|
|
||||||
|
**Open.** `96eb95ce12f2` (*Mnemosyne Watchdog*, 15-minute cadence) has failed **every slot** since
|
||||||
|
**2026-11-18 20:01:06 AWST** — **115 consecutive failures** through **2026-11-20 00:31:16 AWST**, the last
|
||||||
|
recorded before this page was written (it was 114 when the audit began; the streak gained a slot mid-audit).
|
||||||
|
`cron/jobs.json` reads `last_status: error`, **`failure_streak: 115`**.
|
||||||
|
|
||||||
|
This is the estate's **only** red cron row with a large streak, and it is a *single durable condition*
|
||||||
|
re-reported every 15 minutes, not a series of separate faults.
|
||||||
|
|
||||||
|
## The failure is a fixed, self-describing threshold trip
|
||||||
|
|
||||||
|
```
|
||||||
|
Script exited with code 1
|
||||||
|
stdout:
|
||||||
|
🚨 Many unresolved conflicts: 59 (threshold: 50)
|
||||||
|
```
|
||||||
|
|
||||||
|
`cron_incidents` row **`96eb95_98ade422d195`** — `state: alerted`, `failure_type: script`,
|
||||||
|
`first_seen_at 2026-11-18T20:01:15.906837+08:00`, `last_seen_at 2026-11-20T00:31:16.326357+08:00`,
|
||||||
|
`alerted_at 2026-11-19T20:46:23.428143+08:00`, `closed_at: None`, output file
|
||||||
|
`/home/hermes/.hermes/cron/output/96eb95ce12f2/2026-11-20_00-16-15.md`.
|
||||||
|
|
||||||
|
Unlike the health monitor's rows (whose signatures embed a changing float and so fragment into a new row per
|
||||||
|
slot → [[incidents/2026-11-14-mnemosyne-capture-stall]]), **this one fixes on the constant count `59` and
|
||||||
|
therefore stays a single row.** The cost of that stability is the mirror image: it never closes, and it
|
||||||
|
re-alerts forever.
|
||||||
|
|
||||||
|
## Alerting works — and that is the problem
|
||||||
|
|
||||||
|
`deliveries.db`, window 2026-11-18 → 2026-11-20 00:0x — the watchdog's own failure notices:
|
||||||
|
|
||||||
|
| delivered | execution |
|
||||||
|
|---|---|
|
||||||
|
| 2026-11-18 20:01:16 | `f9865723b4f64d9fbfca7711ab053056` |
|
||||||
|
| 2026-11-19 02:16:16 | `7aa35681fd914a1e8db64b8dd0e09c84` |
|
||||||
|
| 2026-11-19 08:31:17 | `1ccc7e280d234287973edf5e0b35a5b7` |
|
||||||
|
| 2026-11-19 12:01:59 | `396940d9514b419eb024b83b3c10198d` |
|
||||||
|
| 2026-11-19 14:31:35 | `0766953f92dc4fc4b0d0404dca51683f` |
|
||||||
|
| 2026-11-19 20:46:19 | `db010a8294bf4cd78457a265a5fb167c` |
|
||||||
|
| 2026-11-20 00:03:37 | `64c6b5202463487c906f860bb746138b` |
|
||||||
|
|
||||||
|
All `status: delivered`, all `for_failure: 1`. So detection, the failure notice and the push path are **all
|
||||||
|
working**. The output is an alert roughly every 3–6 h for a threshold breach that has not moved in two days —
|
||||||
|
which is exactly how a red signal stops meaning anything: while this row is red, a *new* Mnemosyne fault
|
||||||
|
arriving on the same job is indistinguishable from the standing one.
|
||||||
|
|
||||||
|
## What the 59 conflicts actually are
|
||||||
|
|
||||||
|
Live read of `~/.hermes/mnemosyne/data/mnemosyne.db` (2026-11-20 00:3x AWST): **223 conflicts total, 59
|
||||||
|
unresolved.** Unresolved by creation day:
|
||||||
|
|
||||||
|
| created | unresolved |
|
||||||
|
|---|---|
|
||||||
|
| 2026-10-31 | 11 |
|
||||||
|
| 2026-11-02 | 2 |
|
||||||
|
| 2026-11-03 | 12 |
|
||||||
|
| 2026-11-06 | 1 |
|
||||||
|
| 2026-11-07 | 2 |
|
||||||
|
| 2026-11-08 | 1 |
|
||||||
|
| 2026-11-12 | 13 |
|
||||||
|
| 2026-11-13 | 3 |
|
||||||
|
| **2026-11-18** | **14** |
|
||||||
|
| **total** | **59** |
|
||||||
|
|
||||||
|
Two things stand out:
|
||||||
|
|
||||||
|
1. **Most of the backlog is old.** 42 of the 59 predate 2026-11-14; the oldest batch is from **2026-10-31**,
|
||||||
|
which is the same batch recorded on [[systems/mnemosyne]] as *"all created in one batch at 2026-10-31
|
||||||
|
19:26:39 and sharing a single `fact_a_id`, the extraction-noise signature"*. The documented remediation for
|
||||||
|
this class is to bulk-resolve them as `false_positive` (see the `mnemosyne-operations` skill) — **not
|
||||||
|
applied here**.
|
||||||
|
2. **The threshold has been raised before.** The 2026-10-18 and 2026-11-01 reads on [[systems/mnemosyne]] both
|
||||||
|
record *"1 unresolved conflict (benign, threshold 10)"*. It now reads **50**. So the tripwire was already
|
||||||
|
loosened 5×, and the backlog has since grown past even that. **UNVERIFIED:** when the threshold was changed
|
||||||
|
from 10 to 50, and by whom; basis: two dated page reads vs the live alert text — the config line itself was
|
||||||
|
not opened in this window.
|
||||||
|
|
||||||
|
## The 2026-11-18 batch coincides with the migration session
|
||||||
|
|
||||||
|
The 14 conflicts created on **2026-11-18** are the newest batch, and 2026-11-18 is also the day of the only
|
||||||
|
interactive session in this window — `20261118_191808_5c9e08` (desktop, 9 messages, title *"Helping migrate
|
||||||
|
context AI assistant"*), whose turns were captured into `working_memory` at **19:25:03 AWST** and summarised
|
||||||
|
episodically at **19:55:20 AWST**. Consistent with the standing assessment that most conflicts in this bank are
|
||||||
|
LLM-extraction artefacts rather than real contradictions, but **UNVERIFIED:** that the 14 rows are extraction
|
||||||
|
noise from that session's turns specifically; basis: same-day coincidence plus the page's prior noise analysis —
|
||||||
|
the fact pairs were not inspected.
|
||||||
|
|
||||||
|
## Not applied / open
|
||||||
|
|
||||||
|
- **Nothing was resolved, reclassified or deleted.** No conflict row was touched; no threshold was changed.
|
||||||
|
- The watchdog's cadence, signature construction and alert path were left as-is.
|
||||||
|
- Open question: should a *standing* threshold breach alert on state change rather than on every slot?
|
||||||
|
Same family as the zero-item-consolidation streak problem on
|
||||||
|
[[incidents/2026-11-14-mnemosyne-capture-stall]] — both are conditions that re-report continuously because
|
||||||
|
the gate tests a level, not a change.
|
||||||
|
|
||||||
|
→ [[systems/mnemosyne]], [[systems/scheduled-tasks]]
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
---
|
||||||
|
title: Wiki Audit failed two consecutive nights on a chained-provider credit/auth fault (2026-11-18, 2026-11-19) — 11-17 and 11-18 unaudited
|
||||||
|
type: incident
|
||||||
|
status: active
|
||||||
|
created: 2026-11-20
|
||||||
|
updated: 2026-11-20
|
||||||
|
verified_on: 2026-11-20
|
||||||
|
review_after: 2026-11-21
|
||||||
|
confidence: high
|
||||||
|
tags: [cron, wiki-audit, incident, model-routing, credentials, credits, router, silent-failure]
|
||||||
|
sources: [live-cron-executions-db, live-cron-incidents, live-jobs-json, live-deliveries-db, live-wiki-log]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Wiki Audit failed two consecutive nights (2026-11-18, 2026-11-19) — 11-17 and 11-18 were never audited
|
||||||
|
|
||||||
|
**Open.** `cbfe434a3b5e` (**Wiki Audit — Nightly**, schedule `0 0 * * *`) failed on **two consecutive slots**,
|
||||||
|
so the audit that should have reviewed **2026-11-17** and the audit that should have reviewed **2026-11-18**
|
||||||
|
never ran. The **2026-11-20 00:00** slot is the first to fire since; at the time of writing its execution row
|
||||||
|
reads `running` (started `2026-11-20T00:00:08`).
|
||||||
|
|
||||||
|
This run therefore audits **three windows** — the standard last 24 h plus the two that were skipped.
|
||||||
|
|
||||||
|
## Evidence (live probes 2026-11-20 00:1x–00:4x AWST)
|
||||||
|
|
||||||
|
| slot | started | finished / last seen | duration | status |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 2026-11-18 00:00 | `2026-11-18T00:00:08` | `2026-11-18T00:01:43` (incident `first_seen_at` `00:01:42.342149+08:00`) | ~1 m 35 s | `failed` |
|
||||||
|
| 2026-11-19 00:00 | `2026-11-19T00:00:16` | `2026-11-19T00:03:02` (incident `last_seen_at` `00:03:01.420319+08:00`) | ~2 m 46 s | `failed` |
|
||||||
|
| 2026-11-20 00:00 | `2026-11-20T00:00:08` | — | — | `running` |
|
||||||
|
|
||||||
|
Both failures carry the **same 314-character `RuntimeError`** — a concatenation of every target the router tried:
|
||||||
|
|
||||||
|
```
|
||||||
|
RuntimeError: command-code/deepseek/deepseek-v4-flash: model — [400]: You have insufficient credits to
|
||||||
|
make this request. Please purchase more credits to continue using the service. (HTTP 400);
|
||||||
|
openai-compatible-chat-conn:52258193/deepseek-v4-flash-0731: auth — [401]: invalid auth token (HTTP 401);
|
||||||
|
openai-compati…
|
||||||
|
```
|
||||||
|
|
||||||
|
The tail is cut off at the 314-char cap, mid-way through a further `openai-compatible-…` target, so the
|
||||||
|
**final** target's verdict is not recoverable from this field. The shape matters more than the wording: this is
|
||||||
|
not one bad pin — the **whole fallback chain** was refused, first on **credit exhaustion** (`command-code`),
|
||||||
|
then on an **invalid auth token** (the second target), then at least one more target whose error is truncated away.
|
||||||
|
|
||||||
|
- `cron/jobs.json`: `last_status: error`, `failure_streak: 2` — clean 0 at the 2026-11-17 00:27 run.
|
||||||
|
- `cron_incidents`: row **`cbfe43_fa12cca1ad4b`** — `state: detected`, `failure_type: auth`,
|
||||||
|
`first_seen_at 2026-11-18T00:01:42.342149+08:00`, `last_seen_at 2026-11-19T00:03:01.420319+08:00`,
|
||||||
|
**`alerted_at: None`**, `acked_at: None`, `closed_at: None`, output file
|
||||||
|
`/home/hermes/.hermes/cron/output/cbfe434a3b5e/2026-11-19_00-03-01.md`.
|
||||||
|
- `deliveries.db`, window since 2026-11-18: **24 rows — 23 `delivered` (9 of them `for_failure: 1`) and one
|
||||||
|
`failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter
|
||||||
|
delivery to telegram`, consistent with *Gmail Action Digest* `08eb86f02d37` reading `delivery_failed`).
|
||||||
|
**No row is attributable to either failed audit slot** — both carry `delivery_outcome: suppressed` on the
|
||||||
|
execution row (as does the clean `2026-11-17` slot), i.e. this job's alerts are not wired to leave the box.
|
||||||
|
So the outage produced **no push to Anthony** and the red cron row was the only signal.
|
||||||
|
|
||||||
|
## Why this matters beyond one job
|
||||||
|
|
||||||
|
The wiki audit is this estate's own feedback loop: its output is the `log.md` entry and the page edits that keep
|
||||||
|
`current-state.md`, `systems/mnemosyne.md` and `systems/scheduled-tasks.md` honest. Two silent skips mean the wiki
|
||||||
|
stood still for two nights while the estate did not — including the window in which the **Mnemosyne watchdog went
|
||||||
|
red for 114 consecutive slots** → [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]]. The gap is
|
||||||
|
visible in `index.md`: its newest feed entry was still **2026-11-16** on the morning of 11-20.
|
||||||
|
|
||||||
|
## Precedent — and a repeating pattern
|
||||||
|
|
||||||
|
Same class as [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]] (**HTTP 429**, *"all targets were
|
||||||
|
skipped by pre-dispatch filters"*, 6 targets evaluated / **0 attempted**) and
|
||||||
|
[[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. Two observations worth carrying forward:
|
||||||
|
|
||||||
|
1. **The 11-10 failure self-cleared on the next slot** — so a single red night for this job is not necessarily a
|
||||||
|
standing fault. This one did **not** self-clear: two slots, identical error, ~1.5 and ~2.7 minutes each.
|
||||||
|
2. **`alerted_at: NULL` on both rows.** The 11-10 row was also left unalerted. Two independent failures of this
|
||||||
|
job have now produced incident rows that never alerted, which points at the job's failure-*delivery* path
|
||||||
|
rather than at the alert throttle.
|
||||||
|
|
||||||
|
## Not applied / open
|
||||||
|
|
||||||
|
- **Nothing was changed**: the job's pin, provider and schedule are untouched, no credential was rotated, no
|
||||||
|
cron entry was edited.
|
||||||
|
- No re-run was forced; the 2026-11-20 00:00 slot was left to fire on schedule (it did, and its result is the
|
||||||
|
entry appended to `log.md` the same night).
|
||||||
|
- **UNVERIFIED:** whether the credit exhaustion is on the `command-code` account specifically or the router's
|
||||||
|
shared pool — the concatenated error does not name which target a *successful* dispatch would have used;
|
||||||
|
basis: the truncated 314-char error string.
|
||||||
|
- **UNVERIFIED:** whether this job's failures are configured `suppressed`, or whether its failure path is simply
|
||||||
|
not wired to delivery; basis: absence of delivery rows, not a read of the job's delivery config.
|
||||||
|
|
||||||
|
→ [[systems/scheduled-tasks]], [[systems/omniroute]]
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-15
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (133)
|
||||||
|
|
||||||
|
- [ 4d] CONTRIBUTING.md
|
||||||
|
- [ 7d] agents/hermes-chaos.md
|
||||||
|
- [ 7d] agents/hermes-production.md
|
||||||
|
- [ 4d] agents/index.md
|
||||||
|
- [ 17d] agents/nanobot.md
|
||||||
|
- [ 4d] agents/open-claw.md
|
||||||
|
- [ 11d] agents/qwenpaw.md
|
||||||
|
- [ 4d] agents/zeroclaw.md
|
||||||
|
- [ 17d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 4d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 4d] comparisons/memory-backends.md
|
||||||
|
- [ 4d] comparisons/model-routers.md
|
||||||
|
- [ 17d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 4d] concepts/agent-memory.md
|
||||||
|
- [ 17d] concepts/browser-automation.md
|
||||||
|
- [ 4d] concepts/model-routing.md
|
||||||
|
- [ 4d] concepts/persistent-shells.md
|
||||||
|
- [ 4d] concepts/tool-calling.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 17d] decisions/index.md
|
||||||
|
- [ 4d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 4d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 17d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 4d] experiments/index.md
|
||||||
|
- [ 3d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 4d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 3d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 8d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 10d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 11d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 3d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 4d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 3d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 11d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 17d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 20d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 20d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 17d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 0d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 14d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 9d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 19d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 5d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 19d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 3d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 0d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 4d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 20d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 20d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 18d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 16d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 16d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 16d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 15d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 16d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 15d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 14d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 14d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 14d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 13d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 12d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 11d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 9d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 9d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 10d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 6d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 10d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 8d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 6d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 4d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 4d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 3d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 2d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 0d] index.md
|
||||||
|
- [ 8d] infrastructure/backups.md
|
||||||
|
- [ 4d] infrastructure/credentials-map.md
|
||||||
|
- [ 8d] infrastructure/docker-services.md
|
||||||
|
- [ 4d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 8d] infrastructure/hosts.md
|
||||||
|
- [ 8d] infrastructure/network-map.md
|
||||||
|
- [ 8d] infrastructure/proxmox.md
|
||||||
|
- [ 3d] infrastructure/tailscale.md
|
||||||
|
- [ 11d] queries/retained-research-answers.md
|
||||||
|
- [ 21d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 14d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 2d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 2d] reports/uber-eats/index.md
|
||||||
|
- [ 4d] runbooks/backup-wiki.md
|
||||||
|
- [ 4d] runbooks/diagnose-docker.md
|
||||||
|
- [ 4d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 8d] runbooks/job-radar.md
|
||||||
|
- [ 4d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 20d] runbooks/provider-health.md
|
||||||
|
- [ 17d] runbooks/recover-docker-service.md
|
||||||
|
- [ 4d] runbooks/restart-browser.md
|
||||||
|
- [ 4d] runbooks/restart-hermes.md
|
||||||
|
- [ 4d] runbooks/restore-openrouter.md
|
||||||
|
- [ 4d] runbooks/test-openrouter.md
|
||||||
|
- [ 4d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 20d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 7d] systems/browser-backend.md
|
||||||
|
- [ 4d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 6d] systems/conduit.md
|
||||||
|
- [ 20d] systems/current-profile.md
|
||||||
|
- [ 4d] systems/guanaco.md
|
||||||
|
- [ 4d] systems/headless-server.md
|
||||||
|
- [ 6d] systems/hermes-agent.md
|
||||||
|
- [ 2d] systems/hermes-hq-push.md
|
||||||
|
- [ 4d] systems/hermes-ui.md
|
||||||
|
- [ 11d] systems/home-assistant.md
|
||||||
|
- [ 16d] systems/homepage-dashboard.md
|
||||||
|
- [ 11d] systems/ignis.md
|
||||||
|
- [ 20d] systems/memory-backend.md
|
||||||
|
- [ 13d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 20d] systems/model-providers.md
|
||||||
|
- [ 11d] systems/nanobot.md
|
||||||
|
- [ 4d] systems/nextcloud.md
|
||||||
|
- [ 4d] systems/obsidian.md
|
||||||
|
- [ 5d] systems/omniroute.md
|
||||||
|
- [ 4d] systems/opencode-go.md
|
||||||
|
- [ 4d] systems/openrouter.md
|
||||||
|
- [ 11d] systems/paseo.md
|
||||||
|
- [ 0d] systems/phonon-tts.md
|
||||||
|
- [ 5d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 4d] systems/terminal-backend.md
|
||||||
|
- [ 4d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 133 fresh, 3 no-date of 136 pages
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-16
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (1)
|
||||||
|
|
||||||
|
- [ 22d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||||
|
|
||||||
|
## FRESH (132)
|
||||||
|
|
||||||
|
- [ 5d] CONTRIBUTING.md
|
||||||
|
- [ 8d] agents/hermes-chaos.md
|
||||||
|
- [ 8d] agents/hermes-production.md
|
||||||
|
- [ 5d] agents/index.md
|
||||||
|
- [ 18d] agents/nanobot.md
|
||||||
|
- [ 5d] agents/open-claw.md
|
||||||
|
- [ 12d] agents/qwenpaw.md
|
||||||
|
- [ 5d] agents/zeroclaw.md
|
||||||
|
- [ 18d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 5d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 5d] comparisons/memory-backends.md
|
||||||
|
- [ 5d] comparisons/model-routers.md
|
||||||
|
- [ 18d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 5d] concepts/agent-memory.md
|
||||||
|
- [ 18d] concepts/browser-automation.md
|
||||||
|
- [ 5d] concepts/model-routing.md
|
||||||
|
- [ 5d] concepts/persistent-shells.md
|
||||||
|
- [ 5d] concepts/tool-calling.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 18d] decisions/index.md
|
||||||
|
- [ 5d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 5d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 18d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 5d] experiments/index.md
|
||||||
|
- [ 4d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 5d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 4d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 9d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 11d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 12d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 4d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 5d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 4d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 12d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 18d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 21d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 21d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 18d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 1d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 15d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 10d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 20d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 6d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 20d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 4d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 1d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 5d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 21d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 21d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 19d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 17d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 17d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 17d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 16d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 17d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 16d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 15d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 15d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 15d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 14d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 13d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 12d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 10d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 10d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 11d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 7d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 11d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 9d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 7d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 5d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 5d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 4d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 3d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 0d] index.md
|
||||||
|
- [ 9d] infrastructure/backups.md
|
||||||
|
- [ 5d] infrastructure/credentials-map.md
|
||||||
|
- [ 9d] infrastructure/docker-services.md
|
||||||
|
- [ 5d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 9d] infrastructure/hosts.md
|
||||||
|
- [ 9d] infrastructure/network-map.md
|
||||||
|
- [ 9d] infrastructure/proxmox.md
|
||||||
|
- [ 4d] infrastructure/tailscale.md
|
||||||
|
- [ 12d] queries/retained-research-answers.md
|
||||||
|
- [ 15d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 15d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 15d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 3d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 3d] reports/uber-eats/index.md
|
||||||
|
- [ 5d] runbooks/backup-wiki.md
|
||||||
|
- [ 5d] runbooks/diagnose-docker.md
|
||||||
|
- [ 5d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 9d] runbooks/job-radar.md
|
||||||
|
- [ 5d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 21d] runbooks/provider-health.md
|
||||||
|
- [ 18d] runbooks/recover-docker-service.md
|
||||||
|
- [ 5d] runbooks/restart-browser.md
|
||||||
|
- [ 5d] runbooks/restart-hermes.md
|
||||||
|
- [ 5d] runbooks/restore-openrouter.md
|
||||||
|
- [ 5d] runbooks/test-openrouter.md
|
||||||
|
- [ 5d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 21d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 8d] systems/browser-backend.md
|
||||||
|
- [ 5d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 7d] systems/conduit.md
|
||||||
|
- [ 21d] systems/current-profile.md
|
||||||
|
- [ 5d] systems/guanaco.md
|
||||||
|
- [ 5d] systems/headless-server.md
|
||||||
|
- [ 7d] systems/hermes-agent.md
|
||||||
|
- [ 3d] systems/hermes-hq-push.md
|
||||||
|
- [ 5d] systems/hermes-ui.md
|
||||||
|
- [ 12d] systems/home-assistant.md
|
||||||
|
- [ 17d] systems/homepage-dashboard.md
|
||||||
|
- [ 12d] systems/ignis.md
|
||||||
|
- [ 21d] systems/memory-backend.md
|
||||||
|
- [ 14d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 21d] systems/model-providers.md
|
||||||
|
- [ 12d] systems/nanobot.md
|
||||||
|
- [ 5d] systems/nextcloud.md
|
||||||
|
- [ 5d] systems/obsidian.md
|
||||||
|
- [ 6d] systems/omniroute.md
|
||||||
|
- [ 5d] systems/opencode-go.md
|
||||||
|
- [ 5d] systems/openrouter.md
|
||||||
|
- [ 12d] systems/paseo.md
|
||||||
|
- [ 1d] systems/phonon-tts.md
|
||||||
|
- [ 6d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 5d] systems/terminal-backend.md
|
||||||
|
- [ 5d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 1 stale, 132 fresh, 3 no-date of 136 pages
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-17
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (7)
|
||||||
|
|
||||||
|
- [ 23d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||||
|
- [ 22d] incidents/2026-10-03-ct450-hermes-update.md (updated 2026-10-26)
|
||||||
|
- [ 22d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md (updated 2026-10-26)
|
||||||
|
- [ 22d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
|
||||||
|
- [ 22d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md (updated 2026-10-26)
|
||||||
|
- [ 22d] runbooks/provider-health.md (updated 2026-10-26)
|
||||||
|
- [ 22d] runbooks/update-hermes-safely.md (updated 2026-10-26)
|
||||||
|
|
||||||
|
## FRESH (127)
|
||||||
|
|
||||||
|
- [ 6d] CONTRIBUTING.md
|
||||||
|
- [ 9d] agents/hermes-chaos.md
|
||||||
|
- [ 9d] agents/hermes-production.md
|
||||||
|
- [ 6d] agents/index.md
|
||||||
|
- [ 19d] agents/nanobot.md
|
||||||
|
- [ 6d] agents/open-claw.md
|
||||||
|
- [ 13d] agents/qwenpaw.md
|
||||||
|
- [ 6d] agents/zeroclaw.md
|
||||||
|
- [ 19d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 6d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 6d] comparisons/memory-backends.md
|
||||||
|
- [ 6d] comparisons/model-routers.md
|
||||||
|
- [ 19d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 6d] concepts/agent-memory.md
|
||||||
|
- [ 19d] concepts/browser-automation.md
|
||||||
|
- [ 6d] concepts/model-routing.md
|
||||||
|
- [ 6d] concepts/persistent-shells.md
|
||||||
|
- [ 6d] concepts/tool-calling.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 19d] decisions/index.md
|
||||||
|
- [ 6d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 6d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 19d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 6d] experiments/index.md
|
||||||
|
- [ 5d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 6d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 5d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 10d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 12d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 13d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 6d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 5d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 13d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 19d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 19d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 16d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 11d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 21d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 7d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 21d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 5d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 2d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 6d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 20d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 18d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 18d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 18d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 17d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 18d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 17d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 16d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 16d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 16d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 15d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 14d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 13d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 11d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 11d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 12d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 8d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 12d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 10d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 8d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 0d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 6d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 6d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 5d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 4d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 0d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||||
|
- [ 0d] index.md
|
||||||
|
- [ 10d] infrastructure/backups.md
|
||||||
|
- [ 6d] infrastructure/credentials-map.md
|
||||||
|
- [ 10d] infrastructure/docker-services.md
|
||||||
|
- [ 6d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 10d] infrastructure/hosts.md
|
||||||
|
- [ 10d] infrastructure/network-map.md
|
||||||
|
- [ 10d] infrastructure/proxmox.md
|
||||||
|
- [ 5d] infrastructure/tailscale.md
|
||||||
|
- [ 13d] queries/retained-research-answers.md
|
||||||
|
- [ 16d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 16d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 16d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 4d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 4d] reports/uber-eats/index.md
|
||||||
|
- [ 6d] runbooks/backup-wiki.md
|
||||||
|
- [ 6d] runbooks/diagnose-docker.md
|
||||||
|
- [ 6d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 10d] runbooks/job-radar.md
|
||||||
|
- [ 6d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 19d] runbooks/recover-docker-service.md
|
||||||
|
- [ 6d] runbooks/restart-browser.md
|
||||||
|
- [ 6d] runbooks/restart-hermes.md
|
||||||
|
- [ 6d] runbooks/restore-openrouter.md
|
||||||
|
- [ 6d] runbooks/test-openrouter.md
|
||||||
|
- [ 6d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 9d] systems/browser-backend.md
|
||||||
|
- [ 6d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 0d] systems/conduit.md
|
||||||
|
- [ 0d] systems/current-profile.md
|
||||||
|
- [ 6d] systems/guanaco.md
|
||||||
|
- [ 6d] systems/headless-server.md
|
||||||
|
- [ 8d] systems/hermes-agent.md
|
||||||
|
- [ 4d] systems/hermes-hq-push.md
|
||||||
|
- [ 6d] systems/hermes-ui.md
|
||||||
|
- [ 13d] systems/home-assistant.md
|
||||||
|
- [ 18d] systems/homepage-dashboard.md
|
||||||
|
- [ 13d] systems/ignis.md
|
||||||
|
- [ 0d] systems/memory-backend.md
|
||||||
|
- [ 15d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 0d] systems/model-providers.md
|
||||||
|
- [ 13d] systems/nanobot.md
|
||||||
|
- [ 6d] systems/nextcloud.md
|
||||||
|
- [ 6d] systems/obsidian.md
|
||||||
|
- [ 7d] systems/omniroute.md
|
||||||
|
- [ 6d] systems/opencode-go.md
|
||||||
|
- [ 6d] systems/openrouter.md
|
||||||
|
- [ 13d] systems/paseo.md
|
||||||
|
- [ 2d] systems/phonon-tts.md
|
||||||
|
- [ 7d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 6d] systems/terminal-backend.md
|
||||||
|
- [ 6d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 7 stale, 127 fresh, 3 no-date of 137 pages
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-18
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (2)
|
||||||
|
|
||||||
|
- [ 24d] reports/communities-wa-social-media-audit-2026.md (updated 2026-10-25)
|
||||||
|
- [ 23d] incidents/2026-10-24-conduit-call-request-refused.md (updated 2026-10-26)
|
||||||
|
|
||||||
|
## FRESH (132)
|
||||||
|
|
||||||
|
- [ 7d] CONTRIBUTING.md
|
||||||
|
- [ 10d] agents/hermes-chaos.md
|
||||||
|
- [ 10d] agents/hermes-production.md
|
||||||
|
- [ 7d] agents/index.md
|
||||||
|
- [ 20d] agents/nanobot.md
|
||||||
|
- [ 7d] agents/open-claw.md
|
||||||
|
- [ 14d] agents/qwenpaw.md
|
||||||
|
- [ 7d] agents/zeroclaw.md
|
||||||
|
- [ 20d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 7d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 7d] comparisons/memory-backends.md
|
||||||
|
- [ 7d] comparisons/model-routers.md
|
||||||
|
- [ 20d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 7d] concepts/agent-memory.md
|
||||||
|
- [ 20d] concepts/browser-automation.md
|
||||||
|
- [ 7d] concepts/model-routing.md
|
||||||
|
- [ 7d] concepts/persistent-shells.md
|
||||||
|
- [ 7d] concepts/tool-calling.md
|
||||||
|
- [ 1d] current-state.md
|
||||||
|
- [ 20d] decisions/index.md
|
||||||
|
- [ 7d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 7d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 20d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 7d] experiments/index.md
|
||||||
|
- [ 6d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 7d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 6d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 11d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 13d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 14d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 6d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 7d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 6d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 14d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 20d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 0d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 0d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 20d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 17d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 12d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 8d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 6d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 3d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 7d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 0d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 21d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 19d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 19d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 19d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 18d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 19d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 18d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 17d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 17d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 17d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 16d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 15d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 14d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 12d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 12d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 13d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 9d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 13d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 11d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 9d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 1d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 7d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 7d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 6d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 5d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 1d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 1d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||||
|
- [ 1d] index.md
|
||||||
|
- [ 11d] infrastructure/backups.md
|
||||||
|
- [ 7d] infrastructure/credentials-map.md
|
||||||
|
- [ 11d] infrastructure/docker-services.md
|
||||||
|
- [ 7d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 11d] infrastructure/hosts.md
|
||||||
|
- [ 11d] infrastructure/network-map.md
|
||||||
|
- [ 11d] infrastructure/proxmox.md
|
||||||
|
- [ 6d] infrastructure/tailscale.md
|
||||||
|
- [ 14d] queries/retained-research-answers.md
|
||||||
|
- [ 17d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 17d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 17d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 5d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 5d] reports/uber-eats/index.md
|
||||||
|
- [ 7d] runbooks/backup-wiki.md
|
||||||
|
- [ 7d] runbooks/diagnose-docker.md
|
||||||
|
- [ 7d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 11d] runbooks/job-radar.md
|
||||||
|
- [ 7d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 0d] runbooks/provider-health.md
|
||||||
|
- [ 20d] runbooks/recover-docker-service.md
|
||||||
|
- [ 7d] runbooks/restart-browser.md
|
||||||
|
- [ 7d] runbooks/restart-hermes.md
|
||||||
|
- [ 7d] runbooks/restore-openrouter.md
|
||||||
|
- [ 7d] runbooks/test-openrouter.md
|
||||||
|
- [ 7d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 0d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 10d] systems/browser-backend.md
|
||||||
|
- [ 7d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 1d] systems/conduit.md
|
||||||
|
- [ 1d] systems/current-profile.md
|
||||||
|
- [ 7d] systems/guanaco.md
|
||||||
|
- [ 7d] systems/headless-server.md
|
||||||
|
- [ 9d] systems/hermes-agent.md
|
||||||
|
- [ 5d] systems/hermes-hq-push.md
|
||||||
|
- [ 7d] systems/hermes-ui.md
|
||||||
|
- [ 14d] systems/home-assistant.md
|
||||||
|
- [ 19d] systems/homepage-dashboard.md
|
||||||
|
- [ 14d] systems/ignis.md
|
||||||
|
- [ 1d] systems/memory-backend.md
|
||||||
|
- [ 16d] systems/messaging-integrations.md
|
||||||
|
- [ 1d] systems/mnemosyne.md
|
||||||
|
- [ 1d] systems/model-providers.md
|
||||||
|
- [ 14d] systems/nanobot.md
|
||||||
|
- [ 7d] systems/nextcloud.md
|
||||||
|
- [ 7d] systems/obsidian.md
|
||||||
|
- [ 8d] systems/omniroute.md
|
||||||
|
- [ 7d] systems/opencode-go.md
|
||||||
|
- [ 7d] systems/openrouter.md
|
||||||
|
- [ 14d] systems/paseo.md
|
||||||
|
- [ 3d] systems/phonon-tts.md
|
||||||
|
- [ 8d] systems/push-notifications.md
|
||||||
|
- [ 1d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 7d] systems/terminal-backend.md
|
||||||
|
- [ 7d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 2 stale, 132 fresh, 3 no-date of 137 pages
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-19
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (134)
|
||||||
|
|
||||||
|
- [ 8d] CONTRIBUTING.md
|
||||||
|
- [ 11d] agents/hermes-chaos.md
|
||||||
|
- [ 11d] agents/hermes-production.md
|
||||||
|
- [ 8d] agents/index.md
|
||||||
|
- [ 21d] agents/nanobot.md
|
||||||
|
- [ 8d] agents/open-claw.md
|
||||||
|
- [ 15d] agents/qwenpaw.md
|
||||||
|
- [ 8d] agents/zeroclaw.md
|
||||||
|
- [ 21d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 8d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 8d] comparisons/memory-backends.md
|
||||||
|
- [ 8d] comparisons/model-routers.md
|
||||||
|
- [ 21d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 8d] concepts/agent-memory.md
|
||||||
|
- [ 21d] concepts/browser-automation.md
|
||||||
|
- [ 8d] concepts/model-routing.md
|
||||||
|
- [ 8d] concepts/persistent-shells.md
|
||||||
|
- [ 8d] concepts/tool-calling.md
|
||||||
|
- [ 2d] current-state.md
|
||||||
|
- [ 21d] decisions/index.md
|
||||||
|
- [ 8d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 8d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 21d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 8d] experiments/index.md
|
||||||
|
- [ 7d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 8d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 7d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 14d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 15d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 7d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 8d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 7d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 15d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 21d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 1d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 1d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 21d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 4d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 18d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 13d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 1d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 9d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 1d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 7d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 4d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 0d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 1d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 0d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 20d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 20d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 20d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 19d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 20d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 19d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 18d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 18d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 18d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 17d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 16d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 15d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 13d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 13d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 14d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 10d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 14d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 12d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 10d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 8d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 8d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 7d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 6d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 2d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 2d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||||
|
- [ 2d] index.md
|
||||||
|
- [ 12d] infrastructure/backups.md
|
||||||
|
- [ 8d] infrastructure/credentials-map.md
|
||||||
|
- [ 12d] infrastructure/docker-services.md
|
||||||
|
- [ 8d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 12d] infrastructure/hosts.md
|
||||||
|
- [ 12d] infrastructure/network-map.md
|
||||||
|
- [ 12d] infrastructure/proxmox.md
|
||||||
|
- [ 7d] infrastructure/tailscale.md
|
||||||
|
- [ 15d] queries/retained-research-answers.md
|
||||||
|
- [ 0d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 18d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 18d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 18d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 6d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 6d] reports/uber-eats/index.md
|
||||||
|
- [ 8d] runbooks/backup-wiki.md
|
||||||
|
- [ 8d] runbooks/diagnose-docker.md
|
||||||
|
- [ 8d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 12d] runbooks/job-radar.md
|
||||||
|
- [ 8d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 1d] runbooks/provider-health.md
|
||||||
|
- [ 21d] runbooks/recover-docker-service.md
|
||||||
|
- [ 8d] runbooks/restart-browser.md
|
||||||
|
- [ 8d] runbooks/restart-hermes.md
|
||||||
|
- [ 8d] runbooks/restore-openrouter.md
|
||||||
|
- [ 8d] runbooks/test-openrouter.md
|
||||||
|
- [ 8d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 1d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 11d] systems/browser-backend.md
|
||||||
|
- [ 8d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 2d] systems/conduit.md
|
||||||
|
- [ 2d] systems/current-profile.md
|
||||||
|
- [ 8d] systems/guanaco.md
|
||||||
|
- [ 8d] systems/headless-server.md
|
||||||
|
- [ 10d] systems/hermes-agent.md
|
||||||
|
- [ 6d] systems/hermes-hq-push.md
|
||||||
|
- [ 8d] systems/hermes-ui.md
|
||||||
|
- [ 15d] systems/home-assistant.md
|
||||||
|
- [ 20d] systems/homepage-dashboard.md
|
||||||
|
- [ 15d] systems/ignis.md
|
||||||
|
- [ 2d] systems/memory-backend.md
|
||||||
|
- [ 17d] systems/messaging-integrations.md
|
||||||
|
- [ 2d] systems/mnemosyne.md
|
||||||
|
- [ 2d] systems/model-providers.md
|
||||||
|
- [ 15d] systems/nanobot.md
|
||||||
|
- [ 8d] systems/nextcloud.md
|
||||||
|
- [ 8d] systems/obsidian.md
|
||||||
|
- [ 9d] systems/omniroute.md
|
||||||
|
- [ 8d] systems/opencode-go.md
|
||||||
|
- [ 8d] systems/openrouter.md
|
||||||
|
- [ 15d] systems/paseo.md
|
||||||
|
- [ 4d] systems/phonon-tts.md
|
||||||
|
- [ 9d] systems/push-notifications.md
|
||||||
|
- [ 2d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 8d] systems/terminal-backend.md
|
||||||
|
- [ 8d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 134 fresh, 3 no-date of 137 pages
|
||||||
@@ -0,0 +1,152 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-20
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (136)
|
||||||
|
|
||||||
|
- [ 9d] CONTRIBUTING.md
|
||||||
|
- [ 12d] agents/hermes-chaos.md
|
||||||
|
- [ 12d] agents/hermes-production.md
|
||||||
|
- [ 9d] agents/index.md
|
||||||
|
- [ 0d] agents/nanobot.md
|
||||||
|
- [ 9d] agents/open-claw.md
|
||||||
|
- [ 16d] agents/qwenpaw.md
|
||||||
|
- [ 9d] agents/zeroclaw.md
|
||||||
|
- [ 0d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 9d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 9d] comparisons/memory-backends.md
|
||||||
|
- [ 9d] comparisons/model-routers.md
|
||||||
|
- [ 0d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 9d] concepts/agent-memory.md
|
||||||
|
- [ 0d] concepts/browser-automation.md
|
||||||
|
- [ 9d] concepts/model-routing.md
|
||||||
|
- [ 9d] concepts/persistent-shells.md
|
||||||
|
- [ 9d] concepts/tool-calling.md
|
||||||
|
- [ 0d] current-state.md
|
||||||
|
- [ 0d] decisions/index.md
|
||||||
|
- [ 9d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 9d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 0d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 9d] experiments/index.md
|
||||||
|
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 8d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 13d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 15d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 16d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 9d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 8d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 2d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 10d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 2d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 8d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 5d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 9d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 1d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 2d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 1d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 21d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 21d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 21d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 20d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 21d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 20d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 19d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 19d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 19d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 18d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 17d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 16d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 14d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 14d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 15d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 11d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 15d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 13d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 11d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 3d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 9d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 9d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 8d] incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md
|
||||||
|
- [ 7d] incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md
|
||||||
|
- [ 0d] incidents/2026-11-14-mnemosyne-capture-stall.md
|
||||||
|
- [ 3d] incidents/2026-11-15-plugin-load-timeout-storm.md
|
||||||
|
- [ 0d] incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm.md
|
||||||
|
- [ 0d] incidents/2026-11-18-wiki-audit-credit-outage.md
|
||||||
|
- [ 0d] index.md
|
||||||
|
- [ 13d] infrastructure/backups.md
|
||||||
|
- [ 9d] infrastructure/credentials-map.md
|
||||||
|
- [ 13d] infrastructure/docker-services.md
|
||||||
|
- [ 9d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 13d] infrastructure/hosts.md
|
||||||
|
- [ 13d] infrastructure/network-map.md
|
||||||
|
- [ 13d] infrastructure/proxmox.md
|
||||||
|
- [ 8d] infrastructure/tailscale.md
|
||||||
|
- [ 16d] queries/retained-research-answers.md
|
||||||
|
- [ 1d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 19d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 19d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 19d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 7d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 7d] reports/uber-eats/index.md
|
||||||
|
- [ 9d] runbooks/backup-wiki.md
|
||||||
|
- [ 9d] runbooks/diagnose-docker.md
|
||||||
|
- [ 9d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 13d] runbooks/job-radar.md
|
||||||
|
- [ 9d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 2d] runbooks/provider-health.md
|
||||||
|
- [ 0d] runbooks/recover-docker-service.md
|
||||||
|
- [ 9d] runbooks/restart-browser.md
|
||||||
|
- [ 9d] runbooks/restart-hermes.md
|
||||||
|
- [ 9d] runbooks/restore-openrouter.md
|
||||||
|
- [ 9d] runbooks/test-openrouter.md
|
||||||
|
- [ 9d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 2d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 12d] systems/browser-backend.md
|
||||||
|
- [ 9d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 3d] systems/conduit.md
|
||||||
|
- [ 3d] systems/current-profile.md
|
||||||
|
- [ 9d] systems/guanaco.md
|
||||||
|
- [ 9d] systems/headless-server.md
|
||||||
|
- [ 11d] systems/hermes-agent.md
|
||||||
|
- [ 7d] systems/hermes-hq-push.md
|
||||||
|
- [ 9d] systems/hermes-ui.md
|
||||||
|
- [ 16d] systems/home-assistant.md
|
||||||
|
- [ 21d] systems/homepage-dashboard.md
|
||||||
|
- [ 16d] systems/ignis.md
|
||||||
|
- [ 3d] systems/memory-backend.md
|
||||||
|
- [ 18d] systems/messaging-integrations.md
|
||||||
|
- [ 0d] systems/mnemosyne.md
|
||||||
|
- [ 3d] systems/model-providers.md
|
||||||
|
- [ 16d] systems/nanobot.md
|
||||||
|
- [ 9d] systems/nextcloud.md
|
||||||
|
- [ 9d] systems/obsidian.md
|
||||||
|
- [ 10d] systems/omniroute.md
|
||||||
|
- [ 9d] systems/opencode-go.md
|
||||||
|
- [ 9d] systems/openrouter.md
|
||||||
|
- [ 16d] systems/paseo.md
|
||||||
|
- [ 5d] systems/phonon-tts.md
|
||||||
|
- [ 10d] systems/push-notifications.md
|
||||||
|
- [ 0d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 9d] systems/terminal-backend.md
|
||||||
|
- [ 9d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 136 fresh, 3 no-date of 139 pages
|
||||||
@@ -3,8 +3,8 @@ title: Department of Communities WA Social Media Audit and First-Day Brief
|
|||||||
type: report
|
type: report
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-28
|
created: 2026-08-28
|
||||||
updated: 2026-10-25
|
updated: 2026-11-19
|
||||||
verified_on: 2026-10-25
|
verified_on: 2026-11-19
|
||||||
review_after: 2026-11-26
|
review_after: 2026-11-26
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [content, research, department-of-communities, social-media]
|
tags: [content, research, department-of-communities, social-media]
|
||||||
@@ -37,6 +37,13 @@ point-in-time public-source figures from the 2026-08-28 audit and remain as reco
|
|||||||
not re-counted. The `review_after` date of 2026-11-26 stands for a full refresh against native
|
not re-counted. The `review_after` date of 2026-11-26 stands for a full refresh against native
|
||||||
analytics.
|
analytics.
|
||||||
|
|
||||||
|
**Re-verified 2026-11-19 (stocktake).** Live check of wa.gov.au/organisation/department-of-communities
|
||||||
|
(page last updated 18 November 2026): the remit text is **verbatim-identical** to the quote above,
|
||||||
|
all **five Ministers** are still listed (McGurk, Winton, Beazley, Stojkovski, Swinbourn), and the
|
||||||
|
announcement flow (child-safety package 18 Nov, Seniors Card guide 12 Nov, Seniors Week 11 Nov)
|
||||||
|
confirms the recorded content pillars remain active topics. Structural claims of this page all
|
||||||
|
still hold.
|
||||||
|
|
||||||
## Strategic baseline
|
## Strategic baseline
|
||||||
|
|
||||||
- The Department's current remit spans child protection, community services, homelessness, prevention of family and domestic violence, disability services, early childhood education and care regulation, women's interests, youth, seniors and ageing, carers, and volunteering.
|
- The Department's current remit spans child protection, community services, homelessness, prevention of family and domestic violence, disability services, early childhood education and care regulation, women's interests, youth, seniors and ageing, carers, and volunteering.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Provider Health Check
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-26
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-18
|
||||||
last_tested: 2026-10-26
|
last_tested: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, providers, health, routing]
|
tags: [runbook, providers, health, routing]
|
||||||
@@ -58,6 +58,8 @@ with urllib.request.urlopen(req, timeout=5) as r:
|
|||||||
```
|
```
|
||||||
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
Current verified result (2026-10-26): reachable and returns a non-empty model catalog — **6,480 models** on `omniroute:20128` (CT600, 100.88.81.19).
|
||||||
|
|
||||||
|
> **Re-executed 2026-11-18:** step-3 snippet OK — CT600 `/models` → 200, **6,501 models** (3.6 s; one earlier attempt at the default 5 s timeout timed out, so give this probe headroom). CT600 `/api/health` → 200; `omni-101` `/api/health` → 200; fallback endpoints `opencode.ai` / `inference-api.nousresearch.com` / `ollama.com` all → 200.
|
||||||
|
|
||||||
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
### 3b. Check the default `omni-101` provider (CT101, 100.78.63.20)
|
||||||
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
The **default provider** is `custom:omni-101` → `http://100.78.63.20:20128`, keyed via
|
||||||
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
`HERMES_CUSTOM_100_78_63_20_20128_API_KEY`. Probe it with its **own** key — the CT600 key is rejected here:
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Recover a Docker service
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-29
|
updated: 2026-11-20
|
||||||
verified_on: 2026-10-29
|
verified_on: 2026-11-20
|
||||||
tags: [runbook, docker, recovery]
|
tags: [runbook, docker, recovery]
|
||||||
sources: []
|
sources: []
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -20,6 +20,11 @@ Container shows healthy after recovery probe reboots.
|
|||||||
## Last successfully used
|
## Last successfully used
|
||||||
2026-07-22
|
2026-07-22
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-20 (stocktake, procedure only).** `docker ps` as `hermes` on CT460 still
|
||||||
|
> returns permission denied (`/var/run/docker.sock` is `root:docker 0660`) — sudo or SSH to the
|
||||||
|
> owning host remains required. CT201 homepage degradation **persists**: `:8085` answers **502**
|
||||||
|
> locally, `homepage-manager` up but no dashboard container on 8082. Steel remains fully removed
|
||||||
|
> (re-probed CT201/CT333: no containers, no images, no routes).
|
||||||
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
|
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
|
||||||
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
|
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
|
||||||
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
|
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Update Hermes safely
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-26
|
updated: 2026-11-18
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-18
|
||||||
last_tested: 2026-10-26
|
last_tested: 2026-10-26
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, hermes, update]
|
tags: [runbook, hermes, update]
|
||||||
@@ -28,6 +28,8 @@ Update Hermes to a newer release without losing configuration, skills, or sessio
|
|||||||
- Available disk space in Hermes install and backup paths
|
- Available disk space in Hermes install and backup paths
|
||||||
- No active Hermes chat/webui shell issues before starting
|
- No active Hermes chat/webui shell issues before starting
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-18:** prerequisites hold on CT460 — `hermes --version` → `v0.23.0` from the git install at `/home/hermes/.hermes/hermes-agent`, legacy `venv/` present, three managed runtime venvs present under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. (CT450 currently runs `v0.23.1` — see [[incidents/2026-10-03-ct450-hermes-update]].)
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Create a backup first
|
### 1. Create a backup first
|
||||||
|
|||||||
+15
-2
@@ -3,8 +3,8 @@ title: Conduit
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-10-24
|
created: 2026-10-24
|
||||||
updated: 2026-11-09
|
updated: 2026-11-17
|
||||||
verified_on: 2026-11-09
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-12-08
|
review_after: 2026-12-08
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl, hook, timeout, pre-tool-call]
|
||||||
@@ -255,6 +255,19 @@ The plugin is the **only** `pre_tool_call` registrant on the host (`conduit_push
|
|||||||
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
tool result was still pending"*; *FreshRSS Midday* took **2 h 09 m 53 s**.
|
||||||
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
|
||||||
|
|
||||||
|
### 2026-11-15 — 474 registrations dropped across eight failed loads
|
||||||
|
|
||||||
|
`errors.log` for **2026-11-15** holds **9** × `Failed to load plugin 'conduit_push': load timed out after
|
||||||
|
10s (import + register() never returned)` and **474** × `Plugin 'conduit_push' called
|
||||||
|
register_{tool,hook,skill}() after its load timed out; ignored`, grouped into **eight** load events
|
||||||
|
(01:32:39, 04:47, 06:31, 06:47, 07:06, 08:16, 08:46, 10:47). The loader **discards** late registrations,
|
||||||
|
so for each of those processes **Conduit's tools, skills and hooks were absent** — a silent loss the
|
||||||
|
caller never sees as an error. That same day the `pre_tool_call` hook this plugin registers timed out
|
||||||
|
**11×** (10 tool calls refused). **Zero** such lines on 11-16 and 11-17. Full counts:
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]]; the hook side:
|
||||||
|
[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
|
||||||
|
|
||||||
|
|
||||||
## Open items
|
## Open items
|
||||||
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
- **Hook budget is 120 s, not 30 s — watch it.** A second overrun would eat two minutes per tool call.
|
||||||
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
The binding constraint is `state.db` (1.89 GB, 2,901 sessions, live 40 MB WAL), not this plugin.
|
||||||
|
|||||||
+16
-16
@@ -3,9 +3,9 @@ title: Current Hermes Profile
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-26
|
updated: 2026-11-17
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-17
|
||||||
review_after: 2026-11-09
|
review_after: 2026-12-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, profile]
|
tags: [system, hermes, profile]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026-10-21]
|
||||||
@@ -16,9 +16,9 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
|||||||
## Active Profile
|
## Active Profile
|
||||||
- **Name:** `default`
|
- **Name:** `default`
|
||||||
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
- **Location:** implicit — the top-level `~/.hermes/config.yaml`. There is **no** `~/.hermes/profiles/default/` directory (verified 2026-10-02); `~/.hermes/profiles/` holds only named sub-profiles.
|
||||||
- **Other profiles present:** `ops` only (`~/.hermes/profiles/ops/`) — the Tusk elephant persona, with its own SOUL.md, skills, plugins and memories (last modified 2026-09-30). Other profiles do **not** affect this session; see the profile-isolation note below.
|
- **Other profiles present:** none — `~/.hermes/profiles/` holds only a `.deleted/` marker. The `ops` (Tusk elephant) profile was **deleted 2026-10-11**; two chats still routed to it afterwards, so expect stray references. Other profiles do **not** affect this session; see the profile-isolation note below.
|
||||||
|
|
||||||
## Key Settings (model routing + auxiliary slots re-verified against live config 2026-10-26; remaining items as recorded 2026-10-02)
|
## Key Settings (re-verified against live config 2026-11-17)
|
||||||
|
|
||||||
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
|
### ⚠ Model default moved BACK ONTO OmniRoute (reverted 2026-10-24 19:06)
|
||||||
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
|
- **Provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`),
|
||||||
@@ -38,24 +38,24 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-config, live-config-2026
|
|||||||
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
|
`kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6-1t`.
|
||||||
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
|
The 2026-10-21 record that *no* fallback chain exists is **stale**. ⚠ The first entry is the **dead `nous`
|
||||||
credential** — a fallover would try it first.
|
credential** — a fallover would try it first.
|
||||||
- **⚠ Cron pins did NOT need to move.** **18 of the 28 scheduled jobs pin OmniRoute explicitly**
|
- **⚠ Cron pins (re-verified live from `jobs.json` 2026-11-17).** **21 of the 26 enabled jobs pin OmniRoute
|
||||||
(`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The 8 unpinned jobs follow
|
explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1); the other 5 enabled jobs carry no
|
||||||
the global default, which is OmniRoute again — so the fleet is back to a **single upstream path**, not the
|
pin (no-agent/script jobs) and follow the global default, which is OmniRoute — the fleet runs a **single
|
||||||
two-path split recorded on 2026-10-21. → [[systems/scheduled-tasks]]
|
upstream path**. → [[systems/scheduled-tasks]]
|
||||||
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
|
- **Routing posture:** top-level `fallback_providers` chain + per-job pins are the routing controls;
|
||||||
`model.fallbacks` does not exist.
|
`model.fallbacks` does not exist.
|
||||||
- ⚠ The `custom:omni` provider key is **no longer configured** (only 2 jobs still name it) — see
|
- ⚠ The `custom:omni` provider key is **no longer configured** and **no jobs name it any more** (verified from
|
||||||
[[systems/model-providers]].
|
`jobs.json` 2026-11-17; two jobs still did on 2026-10-26) — see [[systems/model-providers]].
|
||||||
|
|
||||||
### Other settings (recorded 2026-10-02 — auxiliary line re-verified 2026-10-26)
|
### Other settings (re-verified live 2026-11-17)
|
||||||
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13 — previously vision was `stepfun/step-3.7-flash:free` on `nous`. → [[systems/model-providers]]
|
- **Auxiliary slots (all 8 — `approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision`):** `no-think/freeapi/auto` on `custom:omni-101`; per-slot fallbacks `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`; `free_only: true`; compression timeout 90 s, vision timeout 180 s. Re-placed 2026-10-25 19:13. A ninth slot, **`pulse_memory`**, now points at `nous` / `anthropic/claude-haiku-4.5` — i.e. the **dead nous credential**. → [[systems/model-providers]]
|
||||||
- **Switch persistence:** `persist_switch_by_default: false`
|
- **Switch persistence:** `persist_switch_by_default: false`
|
||||||
- **Timezone:** `Australia/Perth`
|
- **Timezone:** `Australia/Perth`
|
||||||
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
- **Approvals mode:** `off` (`destructive_slash_confirm: false`, `mcp_reload_confirm: false`)
|
||||||
- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
|
- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
|
||||||
- **Config version:** `_config_version: 46`; config file ≈514 KB / ≈13,200 lines
|
- **Config version:** `_config_version: 52` (config file ≈514 KB / ≈13,200 lines as of 2026-10-26)
|
||||||
- **MCP servers:** 5 configured
|
- **MCP servers:** 6 configured (`Not human search`, `asana` (disabled), `cua-driver`, `hound`, `mcp-portal`, `approved_shared_memory`)
|
||||||
- **Plugins enabled:** 11
|
- **Plugins enabled:** 15
|
||||||
- **Pet:** configured slug `rhino`, but `enabled: false` (render mode `unicode`, scale 0.55)
|
- **Pet:** configured slug `rhino`, but `enabled: false` (render mode `unicode`, scale 0.55)
|
||||||
- **Skin:** `slate`
|
- **Skin:** `slate`
|
||||||
- **Personality:** empty string in config — the active persona comes from `SOUL.md`, not this key
|
- **Personality:** empty string in config — the active persona comes from `SOUL.md`, not this key
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Memory Backend
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-26
|
updated: 2026-11-17
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-17
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, agent]
|
tags: [memory, mnemosyne, agent]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -34,7 +34,7 @@ Persist and recall agent memory across sessions.
|
|||||||
- **Sleep threshold:** 50 turns
|
- **Sleep threshold:** 50 turns
|
||||||
- **Nudge interval:** 10
|
- **Nudge interval:** 10
|
||||||
- **Ignore patterns:** none set
|
- **Ignore patterns:** none set
|
||||||
- **Live totals (2026-10-26):** 2,350 memories — 842 working / 1,090 episodic / 418 `memoria_facts`
|
- **Live totals (2026-11-17, sqlite):** 2,901 memories — 1,130 working / 1,249 episodic / 522 `memoria_facts`
|
||||||
|
|
||||||
## Storage Location
|
## Storage Location
|
||||||
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
||||||
@@ -48,7 +48,7 @@ Persist and recall agent memory across sessions.
|
|||||||
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||||
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
- **Vector recall degraded silently until 2026-10-03** — `fastembed` was missing from every uv-managed runtime, so `mnemosyne_recall` fell back to FTS5 keyword scoring with no error raised. Repaired 2026-10-03; the 15-min dependency guard now asserts `fastembed` and the model cache path. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
|
||||||
- **Embedding coverage re-verified 2026-10-26** — `check_embedding_coverage.py` reports 1,899 / 2,350 (80.8 %) embedded; **working 815 / 842 (96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)**. Both tiers are **above the 95 % reindex threshold**, so no reindex was triggered. `fastembed` 0.8.1 is present in **both** managed runtimes and 0.8.0 in the legacy venv; `FASTEMBED_CACHE_PATH` is pinned. The 2026-10-03 follow-up (assert `fastembed` in *all* managed runtimes) is now satisfied.
|
- **Embedding coverage re-verified 2026-11-17** — `check_embedding_coverage.py` reports 2,379 / 2,901 (82.0 %) embedded; **working 1,130 / 1,130 (100 %)** and **episodic 1,249 / 1,249 (100 %)** (the gap is `memoria_facts`, unvectorized by design). Both tiers are **above the 95 % reindex threshold**, so no reindex is needed. Earlier (2026-10-26) check: 1,899 / 2,350 (80.8 %), working 96.8 %, episodic 99.4 %.
|
||||||
|
|
||||||
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -3,8 +3,8 @@ title: Model Providers
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-26
|
updated: 2026-11-17
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-17
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [model, provider, inference, routing]
|
tags: [model, provider, inference, routing]
|
||||||
sources: [live-probe, config, web-search]
|
sources: [live-probe, config, web-search]
|
||||||
@@ -20,7 +20,7 @@ Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
|
|||||||
|
|
||||||
| Provider | URL | Models | Role |
|
| Provider | URL | Models | Role |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 18 of 28 cron jobs reference it (2026-10-26) |
|
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 21 of 26 enabled jobs pin it (2026-11-17) |
|
||||||
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
||||||
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
||||||
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
||||||
@@ -53,13 +53,15 @@ Provider names that appear in job configs but not in `custom_providers` do **not
|
|||||||
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
||||||
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
||||||
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
||||||
- **18 of 28 scheduled jobs pin OmniRoute explicitly** (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2). The remaining 8 follow the global default, which is **also** OmniRoute — so the fleet is back to a **single upstream path**. → [[systems/scheduled-tasks]]
|
- **21 of 26 enabled scheduled jobs pin OmniRoute explicitly** (`smart-route` ×14, `free-stack` ×6, `deepseek-paid-smart` ×1; verified from live `jobs.json` 2026-11-17). The remaining 5 enabled jobs carry no pin (no-agent/script jobs) and follow the global default, which is **also** OmniRoute — the fleet runs a **single upstream path**. The two jobs that still named the removed `custom:omni` key on 2026-10-26 no longer do. → [[systems/scheduled-tasks]]
|
||||||
|
|
||||||
## Auxiliary Role Routing (re-verified live 2026-10-26)
|
## Auxiliary Role Routing (re-verified live 2026-11-17)
|
||||||
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
|
**All eight `auxiliary:` slots** — `approval`, `compression`, `curator`, `mcp`, `profile_describer`,
|
||||||
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
|
`skills_hub`, `title_generation`, `vision` — are **`no-think/freeapi/auto` on `custom:omni-101`**, each with
|
||||||
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
|
the chain `no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free` and `free_only: true`.
|
||||||
**Compression timeout 90 s; vision timeout 180 s.**
|
**Compression timeout 90 s; vision timeout 180 s.** A ninth slot, **`pulse_memory`**, is
|
||||||
|
**`nous` / `anthropic/claude-haiku-4.5`** — i.e. it sits on the **dead `nous` credential**; a pulse-memory call
|
||||||
|
that hits it will fail until the slot or the credential is fixed.
|
||||||
|
|
||||||
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
|
- Re-placed **2026-10-25 19:13 AWST**, replacing the `nous` / `stepfun/step-3.7-flash:free` and `custom:omni`
|
||||||
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
|
aux paths recorded on 2026-10-02 and 2026-10-25 (17:50).
|
||||||
|
|||||||
+58
-19
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-15
|
updated: 2026-11-20
|
||||||
verified_on: 2026-11-15
|
verified_on: 2026-11-20
|
||||||
review_after: 2026-11-15
|
review_after: 2026-11-21
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||||
@@ -14,6 +14,17 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
|
- **Current as of 2026-11-20 00:5x (three-window audit — 2026-11-17, 11-18 and 11-19; the audits slotted at 11-18 00:00 and 11-19 00:00 both died, so 11-17 and 11-18 had never been audited; live `jobs.json` + `executions.db` + `deliveries.db` + `cron_incidents` + `mnemosyne.db`).** Ledger: **11-17 — 173 rows, 171 `completed` / 2 `failed`** (both `ea79a2107e36` *Mnemosyne Health Monitor*, `00:01` and `12:01`); **11-18 — 172 rows, 153 / 19** (this audit `00:00:08 → 00:01:43`, the monitor twice, and **16** *Mnemosyne Watchdog* slots from `20:01:06`); **11-19 — 176 rows, 78 / 98** — **the worst day in the retained window** (audit `00:00:16 → 00:03:02`, monitor `12:01`, **96** watchdog slots); **11-20 so far — 8 rows, 3 `completed` / 4 `failed` / 1 `running`** (this audit). Deliveries since 11-18: **24 — 23 `delivered` (9 `for_failure: 1`) / 1 `failed`** (`2026-11-19 08:34:13`, a *content* delivery: `live adapter send failed: Timed out; live adapter delivery to telegram`). Fleet **29 jobs (26 enabled / 3 disabled)** — a **twelfth consecutive window at the same count** (following the 11th, recorded on 11-17; the 11-18 and 11-19 windows went unrecorded because both audits died) — but non-clean `last_status` **1 → 4**: `cbfe434a3b5e` Wiki Audit `error`, `failure_streak` **2** (last clean run 11-17), `96eb95ce12f2` Watchdog `error`, streak **115**, `ea79a2107e36` Health Monitor `error`, streak **2**, `08eb86f02d37` Gmail Action Digest `delivery_failed`, streak **0**. **Two of the three jobs that keep the fleet safe are themselves red — and the 03:00 stocktake is not:** `1a8e90ea2271` ran `completed` on 11-17 (**11 m 35 s**), 11-18 (**23 m 07 s**) and 11-19 (**24 m 38 s**), so the estate's structural maintenance kept running while both alarm paths failed. `cron_incidents` unclosed **5 → 4**, none closed automatically: `cbfe43_fa12cca1ad4b` (*auth*, 2026-11-18 00:01:42, **`alerted_at: None`** — the audit's silent skip), `96eb95_98ade422d195` (*script*, 2026-11-18 20:01:15 → last_seen 2026-11-20 00:31:16), `ea79a2_f2be2db3bfce` (2026-11-19 12:01:58) and `ea79a2_aae4d914f3c9` (2026-11-20 00:03:36) — both monitor rows on the same `STALE MEMORY PIPELINE` metric that is now measuring a *quiet estate* rather than a broken capture (see [[systems/mnemosyne]]). Cadence and pins unchanged; this audit still runs `custom:omni-101` / `deepseek-paid-smart`, and the Mnemosyne Watchdog's model/provider fields remain unset (inherits global). Full detail: [[incidents/2026-11-18-wiki-audit-credit-outage]], [[incidents/2026-11-18-mnemosyne-watchdog-conflict-threshold-storm]].
|
||||||
|
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
|
||||||
|
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
|
||||||
|
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
|
||||||
|
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
|
||||||
|
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
|
||||||
|
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
|
||||||
|
failures and **0** hook timeouts, against 52 and 13 the day before — see
|
||||||
|
[[incidents/2026-11-15-plugin-load-timeout-storm]].
|
||||||
|
|
||||||
|
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
||||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||||
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
Fleet **29 (26 enabled / 3 disabled)**, a **ninth consecutive window at the same count** (nothing added,
|
||||||
@@ -24,11 +35,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
|||||||
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
error`, **`failure_streak` 1** — its **00:00:53 slot failed at 00:03:44 on `Script exited with code 1`**
|
||||||
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
("STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59"), the first red row since the
|
||||||
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
Uber Eats job cleared on **2026-11-13**. **Unclosed incident rows 0 -> 1** (**138** total), the row being
|
||||||
`ea79a2_601037516abd` (`state: alerted`, `failure_type: auth` — **mislabelled**, `alerted_at`
|
`ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5; its `failure_type: auth` is **not**
|
||||||
00:03:44.5) → [[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
evidence of a credential fault — this job's twelve earlier rows read `agent` ×3 / `timeout` ×1 /
|
||||||
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — the failure was in
|
`unknown` ×8, and the estate-wide table is 53 `unknown` / 47 `agent` / 15 `timeout`) →
|
||||||
*detection*, not notification: at **12:00:52** the same slot had already seen the identical hole and
|
[[incidents/2026-11-14-mnemosyne-capture-stall]]. The job's own alert **did deliver**
|
||||||
logged it **ℹ️ informational** (`capture delay -589.5 min`), so **~12 h of lead time produced no alarm**.
|
(`deliveries.db` 2026-11-15 row, `for_failure: 1`, `status: delivered`, 00:03:39) — **notification
|
||||||
|
worked, and so did detection; the earlier slots simply had nothing to see.** *(Corrected 2026-11-15
|
||||||
|
01:2x — an earlier revision of this entry claimed the **12:00:52** slot had "already seen the identical
|
||||||
|
hole" and logged it ℹ️ at `-589.5 min`, giving "~12 h of lead time"; that was wrong on both counts. See
|
||||||
|
[[#Known Issues]] for the verified mechanism.)*
|
||||||
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
**2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (that monitor run). Next slot of the monitor:
|
||||||
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
**2026-11-15 12:00** (schedule `0 */12 * * *`) — the natural check for whether capture resumed.
|
||||||
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
- **Current as of 2026-11-13 (window 2026-11-12 23:59 -> 2026-11-13 23:59; nightly audit).** **A third
|
||||||
@@ -311,17 +326,41 @@ artefact, not an outage** — always probe the alias the fleet actually pins, ne
|
|||||||
### Open 2026-11-15
|
### Open 2026-11-15
|
||||||
|
|
||||||
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
- **The Mnemosyne Health Monitor (`ea79a2107e36`) is the fleet's only non-clean row** — `last_status: error`,
|
||||||
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`, wrongly labelled
|
`failure_streak: 1`, unclosed incident `ea79a2_601037516abd` (`state: alerted`; the row's
|
||||||
`failure_type: auth`). Its 00:00 slot exited 1 at **00:03:44** on `STALE MEMORY PIPELINE: capture missed …
|
`failure_type: auth` is the scheduler's classifier and does **not** describe the fault — this job's
|
||||||
last capture 2026-11-14 01:32:59` (raw **UTC** in the alert text, so it *reads* as a 23 h gap; the true
|
earlier rows read `agent` / `timeout` / `unknown`). Its 00:00 slot exited 1 at **00:03:44** on
|
||||||
AWST gap at the alert was **14 h 31 m**). The **12:00:52** slot saw the same hole and logged it
|
`STALE MEMORY PIPELINE: capture missed … last capture 2026-11-14 01:32:59` (raw **UTC** in the alert
|
||||||
**ℹ️ informational** — `capture delay -589.5 min`, the negative sign hiding its growth — so the estate
|
text, so it *reads* as a ~23 h gap; the true AWST gap at the alert was **14 h 31 m**). The alert itself
|
||||||
had **~12 h of lead time and no alarm**. The alert itself delivered (00:03:39, `for_failure: 1`).
|
delivered (00:03:39, `for_failure: 1`). Root cause **not established**; the known
|
||||||
Root cause **not established**; the known stringified-`config.yaml` killer is ruled out. Full page:
|
stringified-`config.yaml` killer is ruled out. Full page:
|
||||||
[[incidents/2026-11-14-mnemosyne-capture-stall]]. **Lessons for this fleet:** (a) a monitor that
|
[[incidents/2026-11-14-mnemosyne-capture-stall]].
|
||||||
distinguishes "capture is late" only in *sign* will never escalate on a growing hole — the threshold must
|
- **Verified mechanism of the escalation (re-derivation pass, 2026-11-15 01:2x AWST — corrects the first
|
||||||
be on magnitude; (b) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
reading):** the **12:00:52** slot did **not** see the hole. `~/.hermes/logs/mnemosyne-health-monitor.log`
|
||||||
misreading trap.
|
pins it by line number — run header `2026-11-14 12:01:52 AWST` at **line 957**, next header
|
||||||
|
(`2026-11-15 00:03:35 AWST`) at **line 1030**, and the `capture delay -589.5 min` line at **line 1046**,
|
||||||
|
i.e. *inside the 00:03:35 run's own block*, immediately above the ❌ at line 1047. The 12:00:52 run's own
|
||||||
|
measurement was `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay **29.4 min**` — healthy, exit 0,
|
||||||
|
delivery `suppressed`. `mnemosyne_freshness_gate.py` is **magnitude-based** (`delay_min < -CAPTURE_LAG_MIN`,
|
||||||
|
`MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min; consolidation age default **36 h**), but it is
|
||||||
|
**activity-aware**: it can only fire once a *conversational* message postdates the last capture. The last
|
||||||
|
captured message before the stall was the 09:03 AWST `ios` turn (captured 09:32:59, +29.4 min); the first
|
||||||
|
message to postdate it was the **19:22 AWST desktop** turn — *after* the 12:00 slot, so midnight was the
|
||||||
|
first slot that could see the miss. **The ~12 h of quiet was the `0 */12` cadence, not a threshold bug.**
|
||||||
|
- **What genuinely went unread — consolidation item counts.** The sleep cycles kept firing every ~31 min and
|
||||||
|
returned **0 items: 29 consecutive zero-item cycles**, streak opening `2026-11-14T02:04:42+00:00`
|
||||||
|
(**10:04:42 AWST**) and running to the alert; the last productive cycles were 4622/4621/4620 at
|
||||||
|
`09:33:34` / `09:33:16` / `09:01:40` (naive **AWST** — see the format trap below). Every run printed
|
||||||
|
`Last consolidation: … (0 items)`, including the healthy 12:00 one, but the gate tests consolidation
|
||||||
|
**age** (`> 36 h`) only — never the item count. A pipeline writing nothing still reads as fresh.
|
||||||
|
**Candidate threshold: an N-consecutive-zero-item-cycle exception.**
|
||||||
|
- **Format trap in `consolidation_log.created_at` (verified this pass):** rows 4640–4651 carry a `+00:00`
|
||||||
|
offset, rows 4620–4622 are **naive and local (AWST)**. Read as one axis they misplace events by 8 h and
|
||||||
|
make the streak look as if it began after the last productive cycle. Same family as the alert text below.
|
||||||
|
- **Lessons for this fleet:** (a) alert timestamps printed from raw UTC inside a +08:00 estate are a standing
|
||||||
|
misreading trap (the alert's `01:32:59` is 09:32:59 AWST); (b) freshness gates must test a pipeline's
|
||||||
|
**output volume**, not just the age of its last run — an empty-but-current cycle is the failure signature
|
||||||
|
for a dead write path; (c) when a ledger's `failure_type` disagrees with the error text, trust the text —
|
||||||
|
the classifier is `unknown`/`agent` for most rows.
|
||||||
|
|
||||||
### Resolved 2026-11-02
|
### Resolved 2026-11-02
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-14
|
updated: 2026-11-20
|
||||||
verified_on: 2026-11-14
|
verified_on: 2026-11-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -12,9 +12,9 @@ sources: []
|
|||||||
|
|
||||||
# Installed Skills
|
# Installed Skills
|
||||||
|
|
||||||
**Total installed: 300 skills** in `~/.hermes/skills/`.
|
**Total installed: 299 skills** in `~/.hermes/skills/`.
|
||||||
|
|
||||||
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
- **51 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||||
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
||||||
|
|
||||||
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||||
@@ -98,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
|||||||
|
|
||||||
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
||||||
|
|
||||||
## Devops (108)
|
## Devops (107)
|
||||||
|
|
||||||
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
||||||
- `agent-extension-audit` — # Agent Extension Audit
|
- `agent-extension-audit` — # Agent Extension Audit
|
||||||
@@ -204,7 +204,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
|||||||
- `uber-eats-monitor` — # Uber Eats Order Monitor
|
- `uber-eats-monitor` — # Uber Eats Order Monitor
|
||||||
- `vision-model-validation` — # Vision Model Validation via OmniRoute
|
- `vision-model-validation` — # Vision Model Validation via OmniRoute
|
||||||
- `watchers` — # Watchers
|
- `watchers` — # Watchers
|
||||||
- `web-content-extraction` — # Web Content Extraction — Fallback Techniques
|
|
||||||
- `weekly-comms-tracker` — # Weekly Communications Tracker
|
- `weekly-comms-tracker` — # Weekly Communications Tracker
|
||||||
- `wiki-maintenance` — # Wiki Maintenance
|
- `wiki-maintenance` — # Wiki Maintenance
|
||||||
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration
|
- `zeroclaw-ops` — # ZeroClaw Operations & Configuration
|
||||||
|
|||||||
Reference in New Issue
Block a user