diff --git a/agents/hermes-chaos.md b/agents/hermes-chaos.md index bd2df73..9d3dcca 100644 --- a/agents/hermes-chaos.md +++ b/agents/hermes-chaos.md @@ -3,8 +3,8 @@ title: Hermes Chaos type: system status: active created: 2026-08-15 -updated: 2026-11-07 -verified_on: 2026-11-07 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [hermes, chaos, ct450, experimental, agent] sources: [live-probe, config-inspection] @@ -28,7 +28,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod ## Current State - **Status:** Active (experimental) -- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]]. +- **Hermes version:** ⚠ **now `v0.23.0`** — the live install stamp `~/.hermes/hermes-agent/install-stamp.json` inside CT450 reads `baseVersion: 0.23.0`, `displayVersion: 0.23.0`, `distance: 0`, `payload: bootstrap` (probed 2026-11-08 00:4x AWST by the nightly audit; `hermes --version` not re-run in that probe). The rest of this bullet is the 2026-11-07 correction, written when CT450 was still `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 then returned **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim finally became true. `hermes update --check` then reported stable **v0.23.0** (`2bd378ed1b8d`) available; CT460 took it on 2026-11-07 and CT450's stamp now reads the same version. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]]. - **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]]. - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) - **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07 diff --git a/agents/hermes-production.md b/agents/hermes-production.md index 36155cb..d9af0e5 100644 --- a/agents/hermes-production.md +++ b/agents/hermes-production.md @@ -3,8 +3,8 @@ title: Hermes Production type: system status: active created: 2026-08-15 -updated: 2026-11-07 -verified_on: 2026-11-07 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [hermes, production, ct460, agent] sources: [live-probe, config-inspection] @@ -27,7 +27,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn ## Current State - **Status:** Active -- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available) +- **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST. +- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Config:** `~/.hermes/config.yaml` diff --git a/current-state.md b/current-state.md index 0239de1..22e9053 100644 --- a/current-state.md +++ b/current-state.md @@ -12,7 +12,49 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv --- # Current State -Updated: 2026-11-07 +Updated: 2026-11-08 + +## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST) + +- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10, + `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot` + stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**. + Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55 + (live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]] +- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in + *after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health + Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance* + (`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on + `lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00** + (`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] +- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and + `1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with + **`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06. +- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`, + 2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not + left dirty. +- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update. + No `browser.*` config key changed. → [[systems/browser-backend]] +- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own + last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. → + [[systems/conduit]] +- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web + routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. → + [[systems/hermes-hq-push]] +- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on + `…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07 + execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]] +- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into + `~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`, + `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew + **46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the + *"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity + 25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]] +- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7 + `delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. → + [[systems/scheduled-tasks]] +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check + found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed. ## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST) diff --git a/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md index e7100ba..309e0cb 100644 --- a/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +++ b/incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md @@ -105,6 +105,19 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c - **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or scrubbed. +## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement) + +- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for + secret-shaped strings (`sk-…`, `ghp_…`, and any ` = <16+ chars>`) returned + **0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret + would still pass. +- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a + `redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective + on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or + delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**. +- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option + executed. Both **Remediation options 2 and 4 above remain open**. + ## See also - [[infrastructure/credentials-map]] - where secrets live, by design diff --git a/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md b/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md new file mode 100644 index 0000000..25cdafd --- /dev/null +++ b/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md @@ -0,0 +1,121 @@ +--- +title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) +type: incident +status: open +created: 2026-11-08 +updated: 2026-11-08 +verified_on: 2026-11-08 +review_after: 2026-11-15 +confidence: high +tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake] +sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json] +--- + +# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) + +**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in +*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain +killed the two long-running cron executions that were still in flight, and both were closed **`failed`** +with `RuntimeError: Cron job '…' lost its durable fire claim ownership`: + +| job | id | run window (ledger) | closed | +|---|---|---|---| +| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` | +| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` | + +systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter +is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22. +**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own. + +## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated) + +| time | event | +|---|---| +| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` | +| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s | +| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** | +| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 | +| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* | +| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` | +| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails | +| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails | +| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` | +| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* | +| 05:01:41 | gateway back up as `pid 2981474` | +| 05:02:22 | both ledger rows finalised `failed` | +| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** | +| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) | +| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) | +| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved | +| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` | + +## Incident-ledger state (live read 2026-11-08 00:0x AWST) + +| id | job | state | first seen | alerted_at | closed | +|---|---|---|---|---|---| +| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — | +| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — | +| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 | +| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* | + +The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no +alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path. + +## Root cause and working hypotheses + +**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string, +the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is +`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires +the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that +remaining work. + +**Not established — the restart's origin.** The request at ~04:29:52 is: +- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and +- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final + step is recorded as **`restarting_gateway` = failed**. + +*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway +runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.** +Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another +component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt +does not begin until **05:05:13**, after the restart was already pending. + +**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted* +their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs +were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger +cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's +2026-11-07 edits are known to exist only because an in-day session committed them +([[systems/scheduled-tasks]]). + +## Impact + +- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken + **7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour + tail ([[systems/mnemosyne]]). +- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`, + `last_status: error`) — no stocktake ran on 2026-11-07. +- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike + 2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]). + +## Verification evidence + + journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10' + /home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table) + /home/hermes/.hermes/conduit-self-update.json + /home/hermes/.hermes/cron/jobs.json (failure_streak, last_status) + +## Open items + +- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or + fix the alerting path that let a `detected` row sit silent. +- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and + whether the 1800 s after-turn wait should refuse to drain long-running cron executions. + +## See also + +- [[systems/scheduled-tasks]] +- [[systems/hermes-agent]] +- [[systems/conduit]] +- [[systems/mnemosyne]] +- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] diff --git a/index.md b/index.md index bcdb41a..35b3c6d 100644 --- a/index.md +++ b/index.md @@ -12,7 +12,9 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-11-07. +> model routing and automation. Last updated: 2026-11-08. +> +> **2026-11-08 (nightly audit — window 2026-11-07) — two runs orphaned mid-restart, and two incident rows that never alerted.** The gateway restart that arrived at **05:01:00** (after a requested restart sat in *after-turn* mode for the full 1800 s) drained two in-flight cron executions; both closed `failed` on `lost its durable fire claim ownership` — *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22, **second consecutive failed day**, `failure_streak: 2`). The restart was requested at ~04:29:52 — **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, 04:27:55→04:32:32, its own ledger recording `restarting_gateway` = *failed*) — working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt starts 05:05:13). Day counts: **163 rows — 159 `completed` / 3 `failed` / 1 `running`**; the third failure is the agent-log watchdog correctly reporting the event 19 s later. ⚠ **Both stocktake incident rows are still `detected` with `alerted_at: null`** — closed 0 → 2, and neither alerted. Elsewhere in the window: **Hermes 0.22.5 → 0.23.0** on CT460 (snapshot stage **skipped again** — still no rollback point), the `camofox` browser backend arrived as an **enabled plugin (1.1.0)**, `conduit_push` self-updated to **0.134.4**, `hermes-hq-push` reinstalled **0.9.27 → 0.9.42** (routes reloaded by the 11:11 restart), and the **OpenCode Go credential is expired** (HTTP 401 `api_key_expired` — configuration alive, credential dead, nothing consuming it yet). New page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. > > **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.) > @@ -62,9 +64,10 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.27**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-03 11:06**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` `plugins.enabled` line **7556**. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. - [[systems/scheduled-tasks]] — **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** -- [[systems/skills-index]] — Installed skills inventory (**302 skills** (**68 categories**) as of the **2026-11-05** stocktake; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (46 archived skills) — the earlier 346/347 counts included it. Freshness at the **2026-11-07** run: **0 stale / 125 fresh / 3 no-date of 128 pages** (`freshness-20261107.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. +- [[systems/skills-index]] — Installed skills inventory (**299 skills** (**67 categories**) as of the **2026-11-08** audit; regenerated daily by the 03:00 job). The generator **excludes `.archive/`** (50 archived skills) — the earlier 346/347 counts included it. ⚠ **2026-11-07 07:43 — four skills moved into `.archive/` in one pass** (`cron-model-repair`, `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`), taking the count **302 → 299** and the parked archive **46 → 50**; the `Cron Model Repair` category went with its only member. **No active `SKILL.md` references any of the four**, so if this was a consolidation the *"absorbed into X"* pointer is missing. One skill appeared in the same window (`microsoft-office`, Productivity 25 → 26). Freshness at the **2026-11-08** run: **0 stale / 126 fresh / 3 no-date of 129 pages** (`freshness-20261108.txt`). Note: `SKILL.md` files on disk still exceed the index by one (300 vs 299) — the generator de-dups one nested duplicate, so **299 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ✅ **2026-11-05 — the generator divergence that made this page flip-flop is FIXED at the root:** two *different* scripts were writing this page (the repo script → 302/68; the stale `~/.hermes/scripts/wiki-maintain.sh` copy → 349/26, because it used `Path.rglob()` and so counted the parked archive), and the nightly audit was pointed at the stale one. The stale copy is now a **delegating wrapper** with no generator logic of its own → [[incidents/2026-11-05-skills-index-generator-divergence]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] - **OPEN.** A restart requested at **~04:29:52** sat in *after-turn* mode for the full **1800 s**, then timed out and drained at **05:00:43/47** — the same `#77184` behaviour as the 10-31 reload, but this time it killed **two** in-flight cron executions: *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42.918 -> 05:02:22.870) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31.515 -> 05:02:22.174), both closed **`failed`** on `lost its durable fire claim ownership`; systemd restarted the unit at **05:01:00** (`restart counter is at 3`). A **third** failure that night was the agent-log ERROR watchdog correctly catching the event 19 s later. The request landed **inside the `conduit_push` self-update window** (04:27:55 -> 04:32:32, its own ledger recording `restarting_gateway` = *failed*) — the **working hypothesis for the requester, not proven**; the 0.22.5 -> 0.23.0 updater is excluded (receipt starts 05:05:13). ⚠ **Closed 0 -> 2 incident rows and neither alerted**: both `1a8e90_*` rows sit `detected` with `alerted_at: null`. The stocktake slot has now failed **two days running** (`failure_streak: 2`). Job Radar ran clean. → [[systems/scheduled-tasks]], [[systems/hermes-agent]], [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **OPEN.** A `direct`-source fire of `5fc292ff0f3b` (08:27:09 -> 13:26:22, 4 h 59 m 13 s, delivered) settled the claim while the scheduler's **own 11:30:32 slot** was still running, so that slot was interrupted at **13:26:38** and closed **`failed`** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` - a **new string variant** and the first claim-loss failure **with no gateway restart involved**. It raised **no `cron_incidents` row**, so a `failed` execution with no alert was the day's only red row; the discarded run left no output file. The 14:30 slot recovered the day in a normal **12 m 27 s** cycle. Candidate fixes (slot suppression when a claim is live, consuming the next slot for out-of-band fires, raising an incident row on discard) are listed on the page and **none was applied**. - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **OPEN.** Live `.env` values were found stored verbatim in **17 rows of `mnemosyne.db`** (`memories` 4, `episodic_memory` 3, `working_memory` 1, `gists` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - per-key counts in the page), reproduced in full by each of the **7 uncompressed rotation snapshots**, and all four values also sit verbatim in the **1.7 GB session store `state.db`**, which is the likely capture origin. The Camofox key is embedded verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and hence in `cron/deliveries.db`. Nothing rotated, deleted or scrubbed; no value recorded. - [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] - `dca8482e4f76` + `85edf6772bca` died 36 s apart on `RuntimeError: Upstream stream failed before completion.` after 2 h 41 m each; Stage 2 recovered the Job Radar's day (5 roles + 1 cover letter). diff --git a/log.md b/log.md index 52c0f49..ba4d5ad 100644 --- a/log.md +++ b/log.md @@ -2330,3 +2330,57 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* `RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership` (failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are included in this commit so the tree is not left dirty. + +## [2026-11-08] audit - window 2026-11-07 + +- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still + open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**, + 0 `for_failure`. +- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight + cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**: + `ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271` + (*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in + *after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd + restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night + (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event + ~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose + own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not + proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13). +- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's: + `1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both + `detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52. +- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`); + the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback + point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live + `pid 3109529`). CT450's install stamp now also reads **0.23.0**. +- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is + no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push` + reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its + routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401 + `api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on + an auth error. +- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for + secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read + as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed. +- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into + `~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`, + `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the + `Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no + absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window + (Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy). +- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`. +- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new), + [[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]], + [[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated), + [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]], + [[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror + (`/home/hermes/wiki`). +- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run + **after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz + rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at + **2026-11-08 00:38:29 AWST**. +- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no + decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.) +- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and + the upstream error code only. + diff --git a/raw/stocktake/freshness-20261108.txt b/raw/stocktake/freshness-20261108.txt new file mode 100644 index 0000000..fab47ea --- /dev/null +++ b/raw/stocktake/freshness-20261108.txt @@ -0,0 +1,142 @@ +# Wiki freshness audit — 2026-11-08 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (126) + +- [ 19d] CONTRIBUTING.md +- [ 0d] agents/hermes-chaos.md +- [ 0d] agents/hermes-production.md +- [ 19d] agents/index.md +- [ 10d] agents/nanobot.md +- [ 19d] agents/open-claw.md +- [ 4d] agents/qwenpaw.md +- [ 19d] agents/zeroclaw.md +- [ 10d] comparisons/camoufox-vs-chrome.md +- [ 19d] comparisons/hermes-vs-nanobot.md +- [ 19d] comparisons/memory-backends.md +- [ 19d] comparisons/model-routers.md +- [ 10d] comparisons/steel-vs-camoufox.md +- [ 19d] concepts/agent-memory.md +- [ 10d] concepts/browser-automation.md +- [ 19d] concepts/model-routing.md +- [ 19d] concepts/persistent-shells.md +- [ 19d] concepts/tool-calling.md +- [ 1d] current-state.md +- [ 10d] decisions/index.md +- [ 19d] decisions/keep-openrouter-as-fallback.md +- [ 19d] decisions/persistent-shell-enabled.md +- [ 10d] decisions/rollback-steel-to-camofox.md +- [ 19d] experiments/index.md +- [ 15d] incidents/2026-07-29-mnemosyne-relay.md +- [ 19d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 15d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 1d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 3d] incidents/2026-08-15-skills-index-drift.md +- [ 4d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 14d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 19d] incidents/2026-08-29-gmail-read-state.md +- [ 18d] incidents/2026-09-02-sftpgo-outage.md +- [ 4d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 10d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 13d] incidents/2026-10-03-ct450-hermes-update.md +- [ 13d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 10d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 15d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 7d] incidents/2026-10-09-memory-recall-relevance.md +- [ 2d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 12d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 20d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 12d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 17d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 14d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 13d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 13d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 13d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 11d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 9d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 9d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 9d] incidents/2026-10-29-conduit-voice-silence.md +- [ 8d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 9d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 8d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 7d] incidents/2026-10-31-asana-access-failures.md +- [ 7d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 7d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 6d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 5d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 4d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 2d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 2d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 3d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 2d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 3d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 1d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 0d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 1d] index.md +- [ 1d] infrastructure/backups.md +- [ 19d] infrastructure/credentials-map.md +- [ 1d] infrastructure/docker-services.md +- [ 19d] infrastructure/domains-and-tunnels.md +- [ 1d] infrastructure/hosts.md +- [ 1d] infrastructure/network-map.md +- [ 1d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 4d] queries/retained-research-answers.md +- [ 14d] reports/communities-wa-social-media-audit-2026.md +- [ 7d] reports/uber-eats/2026-07.md +- [ 7d] reports/uber-eats/2026-08.md +- [ 7d] reports/uber-eats/2026-09.md +- [ 7d] reports/uber-eats/2026-10.md +- [ 7d] reports/uber-eats/index.md +- [ 19d] runbooks/backup-wiki.md +- [ 19d] runbooks/diagnose-docker.md +- [ 19d] runbooks/gateway-resource-resilience.md +- [ 1d] runbooks/job-radar.md +- [ 19d] runbooks/multiplexer-setup.md +- [ 13d] runbooks/provider-health.md +- [ 10d] runbooks/recover-docker-service.md +- [ 19d] runbooks/restart-browser.md +- [ 19d] runbooks/restart-hermes.md +- [ 19d] runbooks/restore-openrouter.md +- [ 19d] runbooks/test-openrouter.md +- [ 19d] runbooks/troubleshoot-tool-loop.md +- [ 13d] runbooks/update-hermes-safely.md +- [ 0d] systems/browser-backend.md +- [ 19d] systems/cloudflare-tunnel.md +- [ 0d] systems/conduit.md +- [ 13d] systems/current-profile.md +- [ 19d] systems/guanaco.md +- [ 19d] systems/headless-server.md +- [ 0d] systems/hermes-agent.md +- [ 0d] systems/hermes-hq-push.md +- [ 19d] systems/hermes-ui.md +- [ 4d] systems/home-assistant.md +- [ 9d] systems/homepage-dashboard.md +- [ 4d] systems/ignis.md +- [ 13d] systems/memory-backend.md +- [ 6d] systems/messaging-integrations.md +- [ 2d] systems/mnemosyne.md +- [ 13d] systems/model-providers.md +- [ 4d] systems/nanobot.md +- [ 19d] systems/nextcloud.md +- [ 19d] systems/obsidian.md +- [ 7d] systems/omniroute.md +- [ 0d] systems/opencode-go.md +- [ 19d] systems/openrouter.md +- [ 4d] systems/paseo.md +- [ 20d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 19d] systems/terminal-backend.md +- [ 19d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 126 fresh, 3 no-date of 129 pages \ No newline at end of file diff --git a/systems/browser-backend.md b/systems/browser-backend.md index 422aa17..a2fb155 100644 --- a/systems/browser-backend.md +++ b/systems/browser-backend.md @@ -3,8 +3,8 @@ title: Browser Backend type: system status: active created: 2026-07-22 -updated: 2026-11-03 -verified_on: 2026-11-03 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [browser, camofox, automation] sources: [live-probe, config, session-history] @@ -16,6 +16,13 @@ sources: [live-probe, config, session-history] Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping. ## Current Configuration (re-verified 2026-10-13) + +> **2026-11-07/08 — the browser backend now ships as a plugin, not a core feature.** The Hermes +> `0.22.5 → 0.23.0` update installed **`camofox` v1.1.0** as an **enabled plugin** at +> `~/.hermes/plugins/camofox/` (`plugin.yaml`, `camofox_tools.py`, `camofox_policy.py`, +> `camofox_vault.py`, `camofox_cli.py`), confirmed live by `hermes plugins list` on 2026-11-07. +> No `browser.*` key was changed in this window — the sections below still describe the running +> configuration ([[systems/hermes-agent]]). - **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note. - **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox` - **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13. diff --git a/systems/conduit.md b/systems/conduit.md index 461ad4c..2fe5ca1 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,9 +3,9 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-11-01 -verified_on: 2026-11-01 -review_after: 2026-11-08 +updated: 2026-11-08 +verified_on: 2026-11-08 +review_after: 2026-12-08 confidence: high tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051] @@ -207,6 +207,29 @@ mode never applies** despite a passing connection test. **4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy. - **Not re-measured:** per-kind event split, health-table freshness, device pairing state. +## 2026-11-08 00:4x AWST — self-updated again to 0.134.4, and its last step failed + +The plugin updated itself in-window and its own ledger records the final step as **failed**: + + ~/.hermes/conduit-self-update.json + {"id": "916d72daeb644b06b5ae1359ddfdd992", "step": "failed", "from_version": "0.122.2", + "version": "0.134.4", "failed_step": "restarting_gateway", + "detail": "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host.", + "started_at": 1793996875.92, "finished_at": 1793997152.39, "pid": 176771, "boot_id": "f292ee44712be8b3"} + +- Window: **2026-11-07 04:27:55 → 04:32:32 AWST** (276 s). `from_version` **0.122.2** (the version this page + recorded on 2026-11-01) → **0.134.4**; `hermes plugins list` and the plugin tree agree. +- `failed_step: restarting_gateway` is literal: the gateway was **not** restarted at 04:32 — `pid 176771` + was still the live gateway at 04:59 and only restarted at **05:01:00** (systemd), then again at 05:43 as + part of the Hermes 0.22.5 → 0.23.0 update ([[systems/hermes-agent]]). +- **Consequence, and the reason this is recorded here:** a restart was requested at ~04:29:52 and held for + the full 1800 s *after-turn* window, then drained — which killed two in-flight cron runs + ([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]). *Working hypothesis, not proven:* the + requester was this self-update. Either way, **a plugin under `~/.hermes/plugins/` cannot restart a + systemd-managed gateway itself**, and it reported failure rather than retrying. +- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on + 2026-11-07. + ## Open items - ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`. diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 6a6819a..6d40eac 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -3,8 +3,8 @@ title: Hermes Agent type: system status: active created: 2026-07-22 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [system, hermes, agent, llm] sources: [live-hermes-version, live-config, live-skill-inventory] @@ -16,7 +16,7 @@ sources: [live-hermes-version, live-config, live-skill-inventory] Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. ## Current Status -- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. +- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. - **Install method:** git - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on. diff --git a/systems/hermes-hq-push.md b/systems/hermes-hq-push.md index df1f969..1f82bf3 100644 --- a/systems/hermes-hq-push.md +++ b/systems/hermes-hq-push.md @@ -3,9 +3,9 @@ title: hermes-hq-push (Hermes HQ iPhone app plugin) type: system status: active created: 2026-11-04 -updated: 2026-11-04 -verified_on: 2026-11-04 -review_after: 2026-12-04 +updated: 2026-11-08 +verified_on: 2026-11-08 +review_after: 2026-12-08 confidence: medium tags: [push, notifications, plugins, ios, desktop, device-access] sources: [live-hermes-plugins-list, live-plugin-manifest, live-config] @@ -15,8 +15,8 @@ sources: [live-hermes-plugins-list, live-plugin-manifest, live-config] ## What it is -`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*) -living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half +`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.42**, author *Hermes HQ*) +living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-07 06:42**, reinstalled 0.9.27 → 0.9.42). It is the host-side half of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal page, and Grok Live voice token minting. @@ -93,6 +93,24 @@ No key material is recorded on this page and none appears in the manifest — th - `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the verification table above). +## 2026-11-07 — reinstalled to v0.9.42, routes picked up by a gateway restart + +- The plugin was **reinstalled to `v0.9.42`** in this window (from `v0.9.27`) and the **gateway was + restarted at 11:11:44 → 11:11:55 AWST** so the new tool routes were registered (live process then + `pid 3109529`). A plugin's routes are only registered at gateway start, so a reinstall without a restart + changes nothing on the wire — the restart is the delivery step, not a symptom + ([[systems/hermes-agent]]). +- Same night, the plugin tree's mtime moved from **2026-11-03 11:06** (the mtime this page recorded on + 2026-11-04) to **2026-11-07 06:42**; `hermes plugins list` on 2026-11-07 reports `hermes-hq-push` **0.9.42** + enabled (`plugin.yaml` agrees). Installed from `mrcharlesiv/hermes-hq-plugins`, upstream **`d26850e7`**, + verified complete at **60/60 files**. Order of events that made the restart necessary: + `hermes-dashboard` started **05:43:04** and `hermes-gateway` **05:37:34** (the 0.22.5 → 0.23.0 update) — + *before* the 06:42 reinstall, so neither running process had the new routes. A verification pass at + 10:00:46 confirmed the plugin files were newer than both service starts, and the single restart followed + at 11:11:44 → 11:11:55 (`pid 3109529`). +- No app-side fault was recorded on 2026-11-07. Push delivery, device access and Grok Live minting were not + re-exercised end-to-end by this audit — the version/route claim is the verified part. + ## Open - `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk diff --git a/systems/opencode-go.md b/systems/opencode-go.md index 8ca957b..94bbaeb 100644 --- a/systems/opencode-go.md +++ b/systems/opencode-go.md @@ -3,8 +3,8 @@ title: OpenCode Go type: system status: active created: 2026-07-29 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [opencode, provider, models, subscription, limits] sources: [https://opencode.ai/docs/go/] @@ -137,6 +137,31 @@ With Go, you pay $10/month and they aim to give you **6x that in usage** ($60). | Qwen3.6 Plus | `qwen3.6-plus` | same | same | | Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible | +## 2026-11-08 00:0x-00:5x AWST — the stored credential is expired (401 `api_key_expired`) + +**Status: `active` as a configuration, `dead` as a credential.** Two independent live probes in this window +both returned **HTTP 401** on the Go endpoint: + + probe 1 (this audit, 2026-11-08 00:2x): zen_go_http=401 + {"error":{"type":"authentication_error","code":"api_key_expired", + "message":"This API key has expired. Create a new API key in OpenCode Console and + replace it in your client or secret store. …"}} + — the same 401 for `https://opencode.ai/zen/go/v1/models`, and `~/.local/share/opencode/auth.json` + still lists an *OpenCode Go api* entry (the CLI's own store). + probe 2 (2026-11-07): provider=opencode-go (fields `key`, `type`) → HTTP 401, body + `{"error":{"type":"authentication_error","code":"api_key_expired", …}}` + +- This is **upstream expiry, not a Hermes fault**: the key was issued by the OpenCode console and has run + out. Hermes is still configured for the provider (`opencode-go/`, per the table above), and + router bridges exist (`ollama-opencode-go-deepseek-v4-flash`, `-glm-5*`, `-grok-4_6`, `-kimi-k2_5`, …). +- **No 2026-11-07 execution failed on an auth error** — the day's three failures were all the gateway + restart drain ([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]), so nothing in the fleet + was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first + use. +- **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes + provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200. + (Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].) + **Config format:** `opencode-go/` (e.g. `opencode-go/kimi-k3`) **Model list API:** `https://opencode.ai/zen/go/v1/models` diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 53f451c..3aefedc 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-07 -verified_on: 2026-11-07 -review_after: 2026-11-08 +updated: 2026-11-08 +verified_on: 2026-11-08 +review_after: 2026-11-09 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] @@ -1155,3 +1155,82 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes: `log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions). - **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no decision reversed. + +## Re-verified 2026-11-08 (window 2026-11-07, AWST; live reads 2026-11-08 00:0x-00:5x) + +### The day + +- **2026-11-07 settled 163 ledger rows — 159 `completed` / 3 `failed` / 1 `running`** (the *running* row is + this audit, `cbfe434a3b5e`, fired 23:59:58). Same fleet as the previous five audits: **29 jobs, 26 enabled**. +- **Deliveries: 7 rows / 7 `delivered` / 0 `for_failure`** (10 on each of 11-05 and 11-06 — no delivery fault + to explain). + +### The window's failure: a restart drain orphaned two in-flight runs + +- Two of the three `failed` rows are **one event**. `ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → + 05:02:22.870) and `1a8e90ea2271` (*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174) both closed with + `RuntimeError: Cron job '…' lost its durable fire claim ownership`. The drain happened at **05:00:43/47**; + systemd restarted the gateway at **05:01:00** (`Scheduled restart job, restart counter is at 3.`) after a + requested restart sat in *after-turn* mode for the full **1800 s** (`#77184`). +- The restart was requested at **~04:29:52** — 2 s before both *restart-safe cron workers* exited status 1 + (*"after adopting this execution, without writing a durable terminal state; whether side effects ran is + unknown"*, logged 04:29:54) — i.e. **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`, + 04:27:55 → 04:32:32, its own ledger recording `restarting_gateway` = **failed**). Working hypothesis, not + proven: the plugin asked for the restart and could not perform it. The 0.22.5 → 0.23.0 updater is excluded + (its receipt starts 05:05:13). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] +- The third `failed` row is **the watchdog working**: `96eb95ce12f2` (05:02:41.105 → 05:02:59.640) exited 1 + having caught the health monitor's `RuntimeError` traceback in `agent.log` — it reported the event ~19 s + after it was written. + +### Still open + +- ⚠ **Unclosed incident rows 0 → 2, and neither alerted.** Both belong to the stocktake job and sit in state + `detected` with **`alerted_at: null`**: `1a8e90_2f370ef764f5` (first seen 2026-11-07T05:02:21.573) and + `1a8e90_628d914f1f97` (2026-11-07T04:29:54.815). The health monitor's two rows were closed 15:43:17 and + resolved 17:08:52 the same day. This is the *red row, no alert* shape the 11-06 audit recorded for Job + Radar, now reproduced on a different job. +- ⚠ **Non-clean `last_status` 1 → 2:** `1a8e90ea2271` is now `error` with **`failure_streak: 2`** — the 03:00 + stocktake slot has failed on consecutive days (11-06 and 11-07) and again left its stocktake edits + uncommitted; the 11-07 in-day entry folded those edits into its own commit so the tree was not left dirty. + The other non-clean row is unchanged: `066bb5191280` (*Uber Eats Monthly Spend Report*), `delivery_failed`, + streak 0, *"configured thread_id 34795 … was not found"*. +- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] stays **OPEN** (untouched in-window). +- **Job Radar ran clean on 11-07** — no slot failed, no claim conflict; the 11-06 concurrency page is + unchanged. + +### Long runs in-window (>90 m; normal cycles are 12 m - 1 h) + +- *Mnemosyne Health Monitor* `ea79a2107e36` — 5 h 01 m before the drain killed it (its two previous slots ran + 7 h 47 m 56 s and 12 h 13 m 54 s without losing the claim). +- *wiki-stocktake-maintenance* `1a8e90ea2271` — 1 h 59 m 51 s before the drain killed it. + +### Skill inventory movement (in-window — 2026-11-07 07:43 AWST) + +- The stocktake's skills-index regeneration now reads **299 skills / 67 categories**, down from **302 / 68**, + because **four skills were moved into `~/.hermes/skills/.archive/`** at **07:43**: `cron-model-repair`, + `fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`. The parked archive grew **46 → 50**, + and the `Cron Model Repair` category vanished with its only member. +- ⚠ **No active skill references any of the four** — a grep of every `SKILL.md` outside `.archive/` returns + no absorbing skill, so if this was a curator consolidation the *"absorbed into X"* pointer is missing. + Their content survives only in the parked copies. +- One skill **appeared**: `microsoft-office` (Productivity 25 → 26), part of a **10-file skill refresh at + 05:36** that lines up with the 0.22.5 → 0.23.0 update's own skill materialisation (the others were + `computer-use`, `claude-design`, `popular-web-designs`, `document-to-action-items`, `meeting-action-items`, + `product-price-monitor`, `weekly-review-planning`, `grounded-citations`, `dogfood`). +- Filesystem vs. page: **300 `SKILL.md` on disk vs. 299 listed** — the generator de-dups one nested + duplicate, so **299** is the unique figure (behaviour unchanged). + +### Freshness / files touched + +- **Freshness re-run:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261108.txt 21` + → **0 stale / 126 fresh / 3 no-date of 129 pages** (report: `raw/stocktake/freshness-20261108.txt`). + Nothing has crossed the 21-day line; the oldest pages are `CONTRIBUTING.md` and `agents/index.md` at 19 d. +- **Pages touched by this audit:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new), + [[systems/hermes-agent]] (version → 0.23.0, restart record), + [[agents/hermes-production]] (version, 11-07 ledger), [[agents/hermes-chaos]] (CT450 stamp now 0.23.0), + [[systems/browser-backend]] (camofox is now a plugin), [[systems/conduit]] (self-update 0.134.4, failed + `restarting_gateway`), [[systems/hermes-hq-push]] (0.9.27 → 0.9.42), [[systems/opencode-go]] (stored key + expired), [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (redaction-off standing + condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both +- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no + decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level. diff --git a/systems/skills-index.md b/systems/skills-index.md index ef14a0c..b1e12bd 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-07 -verified_on: 2026-11-07 +updated: 2026-11-08 +verified_on: 2026-11-08 confidence: high tags: [skills, hermes] sources: [] @@ -12,9 +12,9 @@ sources: [] # Installed Skills -**Total installed: 302 skills** in `~/.hermes/skills/`. +**Total installed: 299 skills** in `~/.hermes/skills/`. -- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. +- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. This page is auto-regenerated from the live skill inventory. Do not hand-edit. @@ -90,10 +90,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `cron-digest-formatting` — # Cron Digest Formatting (Telegram) -## Cron Model Repair (1) - -- `cron-model-repair` — # Cron Model-Pin Repair - ## Data Science (1) - `jupyter-live-kernel` — # Jupyter Live Kernel (hamelnb) @@ -102,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `ddia-systems` — # Designing Data-Intensive Applications Framework -## Devops (109) +## Devops (107) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `agent-extension-audit` — # Agent Extension Audit @@ -127,7 +123,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `docker-management` — # Docker Management - `external-auth-script-integration` — # External Auth Script Integration - `external-skill-integration` — # External Skill Integration -- `fleet-wide-failure-triage` — # Fleet-Wide Failure Triage - `free-web-tools` — # Free Web Search & Extract - `freshrss-intelligence-filter` — # FreshRSS Intelligence Filter - `gcp-vertex-ai-provider` — # GCP Vertex AI Provider @@ -143,7 +138,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `hermes-gateway-config-loading` — # Hermes Gateway Config Loading Pipeline - `hermes-gateway-ops` — # Hermes Gateway Operations & Platform Debugging - `hermes-instance-migration` — # Hermes Instance Migration (audit + curated port) -- `hermes-instance-update` — # Hermes Instance Update Operations - `hermes-mattermost-integration` — # Hermes Mattermost Integration - `hermes-memory-systems` — # Hermes Agent Memory Systems - `hermes-multi-agent` — # Hermes Multi-Agent Architecture @@ -238,7 +232,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `domain-driven-design` — # Domain-Driven Design Framework -## Email (17) +## Email (16) - `agent-email-patterns` — # Agent Email Patterns - `agentmail` — # AgentMail — Agent-Owned Email Inboxes @@ -255,7 +249,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `email-inbox-triage` — # Email Inbox Triage - `gmail-readonly-fetch` — # Gmail Read-Only Fetch (Large Inbox) - `himalaya` — # Himalaya Email CLI -- `mail-lookup` — # Ad-Hoc Mail Lookup - `realtime-email-monitor` — # Real-time Email Monitor (IMAP IDLE) ## Email Himalaya (1) @@ -388,7 +381,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `pragmatic-programmer` — # The Pragmatic Programmer Framework -## Productivity (25) +## Productivity (26) - `airtable` — # Airtable — Bases, Tables & Records - `box` — # Box @@ -398,6 +391,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `google-workspace` — # Google Workspace - `maps` — # Maps Skill - `meeting-action-items` — # Meeting Action Items +- `microsoft-office` — # Microsoft Office Skill - `nano-pdf` — # nano-pdf - `notion` — # Notion - `notion-database-dedup` — # Notion Database Dedup