audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129

This commit is contained in:
Tony0410
2026-11-08 00:41:58 +08:00
parent 0b4ff25fd2
commit 4fd138bb18
15 changed files with 563 additions and 41 deletions
+3 -3
View File
@@ -3,8 +3,8 @@ title: Hermes Chaos
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-11-07 updated: 2026-11-08
verified_on: 2026-11-07 verified_on: 2026-11-08
confidence: high confidence: high
tags: [hermes, chaos, ct450, experimental, agent] tags: [hermes, chaos, ct450, experimental, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -28,7 +28,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
## Current State ## Current State
- **Status:** Active (experimental) - **Status:** Active (experimental)
- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]]. - **Hermes version:** ⚠ **now `v0.23.0`** — the live install stamp `~/.hermes/hermes-agent/install-stamp.json` inside CT450 reads `baseVersion: 0.23.0`, `displayVersion: 0.23.0`, `distance: 0`, `payload: bootstrap` (probed 2026-11-08 00:4x AWST by the nightly audit; `hermes --version` not re-run in that probe). The rest of this bullet is the 2026-11-07 correction, written when CT450 was still `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 then returned **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim finally became true. `hermes update --check` then reported stable **v0.23.0** (`2bd378ed1b8d`) available; CT460 took it on 2026-11-07 and CT450's stamp now reads the same version. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]]. - **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01) - **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07 - **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
+4 -3
View File
@@ -3,8 +3,8 @@ title: Hermes Production
type: system type: system
status: active status: active
created: 2026-08-15 created: 2026-08-15
updated: 2026-11-07 updated: 2026-11-08
verified_on: 2026-11-07 verified_on: 2026-11-08
confidence: high confidence: high
tags: [hermes, production, ct460, agent] tags: [hermes, production, ct460, agent]
sources: [live-probe, config-inspection] sources: [live-probe, config-inspection]
@@ -27,7 +27,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current State ## Current State
- **Status:** Active - **Status:** Active
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available) - **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST.
- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Config:** `~/.hermes/config.yaml` - **Config:** `~/.hermes/config.yaml`
+43 -1
View File
@@ -12,7 +12,49 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
--- ---
# Current State # Current State
Updated: 2026-11-07 Updated: 2026-11-08
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10,
`bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot`
stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**.
Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55
(live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]]
- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in
*after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health
Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance*
(`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on
`lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00**
(`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and
`1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with
**`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06.
- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`,
2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not
left dirty.
- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update.
No `browser.*` config key changed. → [[systems/browser-backend]]
- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own
last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. →
[[systems/conduit]]
- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web
routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. →
[[systems/hermes-hq-push]]
- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on
`…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07
execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]]
- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew
**46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the
*"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity
25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]]
- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7
`delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. →
[[systems/scheduled-tasks]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check
found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST) ## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
@@ -105,6 +105,19 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or - **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
scrubbed. scrubbed.
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
would still pass.
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
executed. Both **Remediation options 2 and 4 above remain open**.
## See also ## See also
- [[infrastructure/credentials-map]] - where secrets live, by design - [[infrastructure/credentials-map]] - where secrets live, by design
@@ -0,0 +1,121 @@
---
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
type: incident
status: open
created: 2026-11-08
updated: 2026-11-08
verified_on: 2026-11-08
review_after: 2026-11-15
confidence: high
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
---
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
| job | id | run window (ledger) | closed |
|---|---|---|---|
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
| time | event |
|---|---|
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
| 05:01:41 | gateway back up as `pid 2981474` |
| 05:02:22 | both ledger rows finalised `failed` |
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
| id | job | state | first seen | alerted_at | closed |
|---|---|---|---|---|---|
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
## Root cause and working hypotheses
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
remaining work.
**Not established — the restart's origin.** The request at ~04:29:52 is:
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
step is recorded as **`restarting_gateway` = failed**.
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
does not begin until **05:05:13**, after the restart was already pending.
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
2026-11-07 edits are known to exist only because an in-day session committed them
([[systems/scheduled-tasks]]).
## Impact
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
tail ([[systems/mnemosyne]]).
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
`last_status: error`) — no stocktake ran on 2026-11-07.
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
## Verification evidence
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
/home/hermes/.hermes/conduit-self-update.json
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
## Open items
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
fix the alerting path that let a `detected` row sit silent.
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
## See also
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
+5 -2
View File
File diff suppressed because one or more lines are too long
+54
View File
@@ -2330,3 +2330,57 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership` `RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are (failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
included in this commit so the tree is not left dirty. included in this commit so the tree is not left dirty.
## [2026-11-08] audit - window 2026-11-07
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
0 `for_failure`.
- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13).
- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52.
- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`);
the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
`pid 3109529`). CT450's install stamp now also reads **0.23.0**.
- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
an auth error.
- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed.
- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy).
- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`.
- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
(`/home/hermes/wiki`).
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
**2026-11-08 00:38:29 AWST**.
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
the upstream error code only.
+142
View File
@@ -0,0 +1,142 @@
# Wiki freshness audit — 2026-11-08
Threshold: >21 days since `updated`
## STALE (0)
## FRESH (126)
- [ 19d] CONTRIBUTING.md
- [ 0d] agents/hermes-chaos.md
- [ 0d] agents/hermes-production.md
- [ 19d] agents/index.md
- [ 10d] agents/nanobot.md
- [ 19d] agents/open-claw.md
- [ 4d] agents/qwenpaw.md
- [ 19d] agents/zeroclaw.md
- [ 10d] comparisons/camoufox-vs-chrome.md
- [ 19d] comparisons/hermes-vs-nanobot.md
- [ 19d] comparisons/memory-backends.md
- [ 19d] comparisons/model-routers.md
- [ 10d] comparisons/steel-vs-camoufox.md
- [ 19d] concepts/agent-memory.md
- [ 10d] concepts/browser-automation.md
- [ 19d] concepts/model-routing.md
- [ 19d] concepts/persistent-shells.md
- [ 19d] concepts/tool-calling.md
- [ 1d] current-state.md
- [ 10d] decisions/index.md
- [ 19d] decisions/keep-openrouter-as-fallback.md
- [ 19d] decisions/persistent-shell-enabled.md
- [ 10d] decisions/rollback-steel-to-camofox.md
- [ 19d] experiments/index.md
- [ 15d] incidents/2026-07-29-mnemosyne-relay.md
- [ 19d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 15d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 1d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 3d] incidents/2026-08-15-skills-index-drift.md
- [ 4d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 14d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 19d] incidents/2026-08-29-gmail-read-state.md
- [ 18d] incidents/2026-09-02-sftpgo-outage.md
- [ 4d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 10d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 13d] incidents/2026-10-03-ct450-hermes-update.md
- [ 13d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 10d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 15d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 7d] incidents/2026-10-09-memory-recall-relevance.md
- [ 2d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 12d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 20d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 12d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 17d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 14d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 13d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 13d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 13d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 11d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 9d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 9d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 9d] incidents/2026-10-29-conduit-voice-silence.md
- [ 8d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 9d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 8d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 7d] incidents/2026-10-31-asana-access-failures.md
- [ 7d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 7d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 6d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 5d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 4d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 2d] incidents/2026-11-03-job-radar-stream-active-timeout.md
- [ 2d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
- [ 3d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
- [ 2d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
- [ 3d] incidents/2026-11-05-skills-index-generator-divergence.md
- [ 1d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
- [ 0d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
- [ 1d] index.md
- [ 1d] infrastructure/backups.md
- [ 19d] infrastructure/credentials-map.md
- [ 1d] infrastructure/docker-services.md
- [ 19d] infrastructure/domains-and-tunnels.md
- [ 1d] infrastructure/hosts.md
- [ 1d] infrastructure/network-map.md
- [ 1d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 4d] queries/retained-research-answers.md
- [ 14d] reports/communities-wa-social-media-audit-2026.md
- [ 7d] reports/uber-eats/2026-07.md
- [ 7d] reports/uber-eats/2026-08.md
- [ 7d] reports/uber-eats/2026-09.md
- [ 7d] reports/uber-eats/2026-10.md
- [ 7d] reports/uber-eats/index.md
- [ 19d] runbooks/backup-wiki.md
- [ 19d] runbooks/diagnose-docker.md
- [ 19d] runbooks/gateway-resource-resilience.md
- [ 1d] runbooks/job-radar.md
- [ 19d] runbooks/multiplexer-setup.md
- [ 13d] runbooks/provider-health.md
- [ 10d] runbooks/recover-docker-service.md
- [ 19d] runbooks/restart-browser.md
- [ 19d] runbooks/restart-hermes.md
- [ 19d] runbooks/restore-openrouter.md
- [ 19d] runbooks/test-openrouter.md
- [ 19d] runbooks/troubleshoot-tool-loop.md
- [ 13d] runbooks/update-hermes-safely.md
- [ 0d] systems/browser-backend.md
- [ 19d] systems/cloudflare-tunnel.md
- [ 0d] systems/conduit.md
- [ 13d] systems/current-profile.md
- [ 19d] systems/guanaco.md
- [ 19d] systems/headless-server.md
- [ 0d] systems/hermes-agent.md
- [ 0d] systems/hermes-hq-push.md
- [ 19d] systems/hermes-ui.md
- [ 4d] systems/home-assistant.md
- [ 9d] systems/homepage-dashboard.md
- [ 4d] systems/ignis.md
- [ 13d] systems/memory-backend.md
- [ 6d] systems/messaging-integrations.md
- [ 2d] systems/mnemosyne.md
- [ 13d] systems/model-providers.md
- [ 4d] systems/nanobot.md
- [ 19d] systems/nextcloud.md
- [ 19d] systems/obsidian.md
- [ 7d] systems/omniroute.md
- [ 0d] systems/opencode-go.md
- [ 19d] systems/openrouter.md
- [ 4d] systems/paseo.md
- [ 20d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 19d] systems/terminal-backend.md
- [ 19d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 0 stale, 126 fresh, 3 no-date of 129 pages
+9 -2
View File
@@ -3,8 +3,8 @@ title: Browser Backend
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-03 updated: 2026-11-08
verified_on: 2026-11-03 verified_on: 2026-11-08
confidence: high confidence: high
tags: [browser, camofox, automation] tags: [browser, camofox, automation]
sources: [live-probe, config, session-history] sources: [live-probe, config, session-history]
@@ -16,6 +16,13 @@ sources: [live-probe, config, session-history]
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping. Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
## Current Configuration (re-verified 2026-10-13) ## Current Configuration (re-verified 2026-10-13)
> **2026-11-07/08 — the browser backend now ships as a plugin, not a core feature.** The Hermes
> `0.22.5 → 0.23.0` update installed **`camofox` v1.1.0** as an **enabled plugin** at
> `~/.hermes/plugins/camofox/` (`plugin.yaml`, `camofox_tools.py`, `camofox_policy.py`,
> `camofox_vault.py`, `camofox_cli.py`), confirmed live by `hermes plugins list` on 2026-11-07.
> No `browser.*` key was changed in this window — the sections below still describe the running
> configuration ([[systems/hermes-agent]]).
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note. - **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox` - **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13. - **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
+26 -3
View File
@@ -3,9 +3,9 @@ title: Conduit
type: system type: system
status: active status: active
created: 2026-10-24 created: 2026-10-24
updated: 2026-11-01 updated: 2026-11-08
verified_on: 2026-11-01 verified_on: 2026-11-08
review_after: 2026-11-08 review_after: 2026-12-08
confidence: high confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
@@ -207,6 +207,29 @@ mode never applies** despite a passing connection test.
**4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy. **4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy.
- **Not re-measured:** per-kind event split, health-table freshness, device pairing state. - **Not re-measured:** per-kind event split, health-table freshness, device pairing state.
## 2026-11-08 00:4x AWST — self-updated again to 0.134.4, and its last step failed
The plugin updated itself in-window and its own ledger records the final step as **failed**:
~/.hermes/conduit-self-update.json
{"id": "916d72daeb644b06b5ae1359ddfdd992", "step": "failed", "from_version": "0.122.2",
"version": "0.134.4", "failed_step": "restarting_gateway",
"detail": "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host.",
"started_at": 1793996875.92, "finished_at": 1793997152.39, "pid": 176771, "boot_id": "f292ee44712be8b3"}
- Window: **2026-11-07 04:27:55 → 04:32:32 AWST** (276 s). `from_version` **0.122.2** (the version this page
recorded on 2026-11-01) → **0.134.4**; `hermes plugins list` and the plugin tree agree.
- `failed_step: restarting_gateway` is literal: the gateway was **not** restarted at 04:32 — `pid 176771`
was still the live gateway at 04:59 and only restarted at **05:01:00** (systemd), then again at 05:43 as
part of the Hermes 0.22.5 → 0.23.0 update ([[systems/hermes-agent]]).
- **Consequence, and the reason this is recorded here:** a restart was requested at ~04:29:52 and held for
the full 1800 s *after-turn* window, then drained — which killed two in-flight cron runs
([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]). *Working hypothesis, not proven:* the
requester was this self-update. Either way, **a plugin under `~/.hermes/plugins/` cannot restart a
systemd-managed gateway itself**, and it reported failure rather than retrying.
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
2026-11-07.
## Open items ## Open items
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean - ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`. at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
+3 -3
View File
@@ -3,8 +3,8 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-30 updated: 2026-11-08
verified_on: 2026-10-30 verified_on: 2026-11-08
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory] sources: [live-hermes-version, live-config, live-skill-inventory]
@@ -16,7 +16,7 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status ## Current Status
- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`. - **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Install method:** git - **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15) - **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on. - **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
+23 -5
View File
@@ -3,9 +3,9 @@ title: hermes-hq-push (Hermes HQ iPhone app plugin)
type: system type: system
status: active status: active
created: 2026-11-04 created: 2026-11-04
updated: 2026-11-04 updated: 2026-11-08
verified_on: 2026-11-04 verified_on: 2026-11-08
review_after: 2026-12-04 review_after: 2026-12-08
confidence: medium confidence: medium
tags: [push, notifications, plugins, ios, desktop, device-access] tags: [push, notifications, plugins, ios, desktop, device-access]
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config] sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
@@ -15,8 +15,8 @@ sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
## What it is ## What it is
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*) `hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.42**, author *Hermes HQ*)
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-07 06:42**, reinstalled 0.9.27 → 0.9.42). It is the host-side half
of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal
page, and Grok Live voice token minting. page, and Grok Live voice token minting.
@@ -93,6 +93,24 @@ No key material is recorded on this page and none appears in the manifest — th
- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the - `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the
verification table above). verification table above).
## 2026-11-07 — reinstalled to v0.9.42, routes picked up by a gateway restart
- The plugin was **reinstalled to `v0.9.42`** in this window (from `v0.9.27`) and the **gateway was
restarted at 11:11:44 → 11:11:55 AWST** so the new tool routes were registered (live process then
`pid 3109529`). A plugin's routes are only registered at gateway start, so a reinstall without a restart
changes nothing on the wire — the restart is the delivery step, not a symptom
([[systems/hermes-agent]]).
- Same night, the plugin tree's mtime moved from **2026-11-03 11:06** (the mtime this page recorded on
2026-11-04) to **2026-11-07 06:42**; `hermes plugins list` on 2026-11-07 reports `hermes-hq-push` **0.9.42**
enabled (`plugin.yaml` agrees). Installed from `mrcharlesiv/hermes-hq-plugins`, upstream **`d26850e7`**,
verified complete at **60/60 files**. Order of events that made the restart necessary:
`hermes-dashboard` started **05:43:04** and `hermes-gateway` **05:37:34** (the 0.22.5 → 0.23.0 update) —
*before* the 06:42 reinstall, so neither running process had the new routes. A verification pass at
10:00:46 confirmed the plugin files were newer than both service starts, and the single restart followed
at 11:11:44 → 11:11:55 (`pid 3109529`).
- No app-side fault was recorded on 2026-11-07. Push delivery, device access and Grok Live minting were not
re-exercised end-to-end by this audit — the version/route claim is the verified part.
## Open ## Open
- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk - `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk
+27 -2
View File
@@ -3,8 +3,8 @@ title: OpenCode Go
type: system type: system
status: active status: active
created: 2026-07-29 created: 2026-07-29
updated: 2026-10-20 updated: 2026-11-08
verified_on: 2026-10-20 verified_on: 2026-11-08
confidence: high confidence: high
tags: [opencode, provider, models, subscription, limits] tags: [opencode, provider, models, subscription, limits]
sources: [https://opencode.ai/docs/go/] sources: [https://opencode.ai/docs/go/]
@@ -137,6 +137,31 @@ With Go, you pay $10/month and they aim to give you **6x that in usage** ($60).
| Qwen3.6 Plus | `qwen3.6-plus` | same | same | | Qwen3.6 Plus | `qwen3.6-plus` | same | same |
| Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible | | Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
## 2026-11-08 00:0x-00:5x AWST — the stored credential is expired (401 `api_key_expired`)
**Status: `active` as a configuration, `dead` as a credential.** Two independent live probes in this window
both returned **HTTP 401** on the Go endpoint:
probe 1 (this audit, 2026-11-08 00:2x): zen_go_http=401
{"error":{"type":"authentication_error","code":"api_key_expired",
"message":"This API key has expired. Create a new API key in OpenCode Console and
replace it in your client or secret store. …"}}
— the same 401 for `https://opencode.ai/zen/go/v1/models`, and `~/.local/share/opencode/auth.json`
still lists an *OpenCode Go api* entry (the CLI's own store).
probe 2 (2026-11-07): provider=opencode-go (fields `key`, `type`) → HTTP 401, body
`{"error":{"type":"authentication_error","code":"api_key_expired", …}}`
- This is **upstream expiry, not a Hermes fault**: the key was issued by the OpenCode console and has run
out. Hermes is still configured for the provider (`opencode-go/<model-id>`, per the table above), and
router bridges exist (`ollama-opencode-go-deepseek-v4-flash`, `-glm-5*`, `-grok-4_6`, `-kimi-k2_5`, …).
- **No 2026-11-07 execution failed on an auth error** — the day's three failures were all the gateway
restart drain ([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]), so nothing in the fleet
was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first
use.
- **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes
provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200.
(Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].)
**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`) **Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
**Model list API:** `https://opencode.ai/zen/go/v1/models` **Model list API:** `https://opencode.ai/zen/go/v1/models`
+82 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-07 updated: 2026-11-08
verified_on: 2026-11-07 verified_on: 2026-11-08
review_after: 2026-11-08 review_after: 2026-11-09
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -1155,3 +1155,82 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions). `log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no - **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. decision reversed.
## Re-verified 2026-11-08 (window 2026-11-07, AWST; live reads 2026-11-08 00:0x-00:5x)
### The day
- **2026-11-07 settled 163 ledger rows — 159 `completed` / 3 `failed` / 1 `running`** (the *running* row is
this audit, `cbfe434a3b5e`, fired 23:59:58). Same fleet as the previous five audits: **29 jobs, 26 enabled**.
- **Deliveries: 7 rows / 7 `delivered` / 0 `for_failure`** (10 on each of 11-05 and 11-06 — no delivery fault
to explain).
### The window's failure: a restart drain orphaned two in-flight runs
- Two of the three `failed` rows are **one event**. `ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 →
05:02:22.870) and `1a8e90ea2271` (*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174) both closed with
`RuntimeError: Cron job '…' lost its durable fire claim ownership`. The drain happened at **05:00:43/47**;
systemd restarted the gateway at **05:01:00** (`Scheduled restart job, restart counter is at 3.`) after a
requested restart sat in *after-turn* mode for the full **1800 s** (`#77184`).
- The restart was requested at **~04:29:52** — 2 s before both *restart-safe cron workers* exited status 1
(*"after adopting this execution, without writing a durable terminal state; whether side effects ran is
unknown"*, logged 04:29:54) — i.e. **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`,
04:27:55 → 04:32:32, its own ledger recording `restarting_gateway` = **failed**). Working hypothesis, not
proven: the plugin asked for the restart and could not perform it. The 0.22.5 → 0.23.0 updater is excluded
(its receipt starts 05:05:13). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- The third `failed` row is **the watchdog working**: `96eb95ce12f2` (05:02:41.105 → 05:02:59.640) exited 1
having caught the health monitor's `RuntimeError` traceback in `agent.log` — it reported the event ~19 s
after it was written.
### Still open
- ⚠ **Unclosed incident rows 0 → 2, and neither alerted.** Both belong to the stocktake job and sit in state
`detected` with **`alerted_at: null`**: `1a8e90_2f370ef764f5` (first seen 2026-11-07T05:02:21.573) and
`1a8e90_628d914f1f97` (2026-11-07T04:29:54.815). The health monitor's two rows were closed 15:43:17 and
resolved 17:08:52 the same day. This is the *red row, no alert* shape the 11-06 audit recorded for Job
Radar, now reproduced on a different job.
- ⚠ **Non-clean `last_status` 1 → 2:** `1a8e90ea2271` is now `error` with **`failure_streak: 2`** — the 03:00
stocktake slot has failed on consecutive days (11-06 and 11-07) and again left its stocktake edits
uncommitted; the 11-07 in-day entry folded those edits into its own commit so the tree was not left dirty.
The other non-clean row is unchanged: `066bb5191280` (*Uber Eats Monthly Spend Report*), `delivery_failed`,
streak 0, *"configured thread_id 34795 … was not found"*.
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] stays **OPEN** (untouched in-window).
- **Job Radar ran clean on 11-07** — no slot failed, no claim conflict; the 11-06 concurrency page is
unchanged.
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
- *Mnemosyne Health Monitor* `ea79a2107e36` — 5 h 01 m before the drain killed it (its two previous slots ran
7 h 47 m 56 s and 12 h 13 m 54 s without losing the claim).
- *wiki-stocktake-maintenance* `1a8e90ea2271` — 1 h 59 m 51 s before the drain killed it.
### Skill inventory movement (in-window — 2026-11-07 07:43 AWST)
- The stocktake's skills-index regeneration now reads **299 skills / 67 categories**, down from **302 / 68**,
because **four skills were moved into `~/.hermes/skills/.archive/`** at **07:43**: `cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`. The parked archive grew **46 → 50**,
and the `Cron Model Repair` category vanished with its only member.
- ⚠ **No active skill references any of the four** — a grep of every `SKILL.md` outside `.archive/` returns
no absorbing skill, so if this was a curator consolidation the *"absorbed into X"* pointer is missing.
Their content survives only in the parked copies.
- One skill **appeared**: `microsoft-office` (Productivity 25 → 26), part of a **10-file skill refresh at
05:36** that lines up with the 0.22.5 → 0.23.0 update's own skill materialisation (the others were
`computer-use`, `claude-design`, `popular-web-designs`, `document-to-action-items`, `meeting-action-items`,
`product-price-monitor`, `weekly-review-planning`, `grounded-citations`, `dogfood`).
- Filesystem vs. page: **300 `SKILL.md` on disk vs. 299 listed** — the generator de-dups one nested
duplicate, so **299** is the unique figure (behaviour unchanged).
### Freshness / files touched
- **Freshness re-run:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261108.txt 21`
→ **0 stale / 126 fresh / 3 no-date of 129 pages** (report: `raw/stocktake/freshness-20261108.txt`).
Nothing has crossed the 21-day line; the oldest pages are `CONTRIBUTING.md` and `agents/index.md` at 19 d.
- **Pages touched by this audit:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
[[systems/hermes-agent]] (version → 0.23.0, restart record),
[[agents/hermes-production]] (version, 11-07 ledger), [[agents/hermes-chaos]] (CT450 stamp now 0.23.0),
[[systems/browser-backend]] (camofox is now a plugin), [[systems/conduit]] (self-update 0.134.4, failed
`restarting_gateway`), [[systems/hermes-hq-push]] (0.9.27 → 0.9.42), [[systems/opencode-go]] (stored key
expired), [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (redaction-off standing
condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both
- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level.
+8 -14
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-07 updated: 2026-11-08
verified_on: 2026-11-07 verified_on: 2026-11-08
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []
@@ -12,9 +12,9 @@ sources: []
# Installed Skills # Installed Skills
**Total installed: 302 skills** in `~/.hermes/skills/`. **Total installed: 299 skills** in `~/.hermes/skills/`.
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them. - **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory. - The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
This page is auto-regenerated from the live skill inventory. Do not hand-edit. This page is auto-regenerated from the live skill inventory. Do not hand-edit.
@@ -90,10 +90,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `cron-digest-formatting` — # Cron Digest Formatting (Telegram) - `cron-digest-formatting` — # Cron Digest Formatting (Telegram)
## Cron Model Repair (1)
- `cron-model-repair` — # Cron Model-Pin Repair
## Data Science (1) ## Data Science (1)
- `jupyter-live-kernel` — # Jupyter Live Kernel (hamelnb) - `jupyter-live-kernel` — # Jupyter Live Kernel (hamelnb)
@@ -102,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `ddia-systems` — # Designing Data-Intensive Applications Framework - `ddia-systems` — # Designing Data-Intensive Applications Framework
## Devops (109) ## Devops (107)
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS) - `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
- `agent-extension-audit` — # Agent Extension Audit - `agent-extension-audit` — # Agent Extension Audit
@@ -127,7 +123,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `docker-management` — # Docker Management - `docker-management` — # Docker Management
- `external-auth-script-integration` — # External Auth Script Integration - `external-auth-script-integration` — # External Auth Script Integration
- `external-skill-integration` — # External Skill Integration - `external-skill-integration` — # External Skill Integration
- `fleet-wide-failure-triage` — # Fleet-Wide Failure Triage
- `free-web-tools` — # Free Web Search & Extract - `free-web-tools` — # Free Web Search & Extract
- `freshrss-intelligence-filter` — # FreshRSS Intelligence Filter - `freshrss-intelligence-filter` — # FreshRSS Intelligence Filter
- `gcp-vertex-ai-provider` — # GCP Vertex AI Provider - `gcp-vertex-ai-provider` — # GCP Vertex AI Provider
@@ -143,7 +138,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `hermes-gateway-config-loading` — # Hermes Gateway Config Loading Pipeline - `hermes-gateway-config-loading` — # Hermes Gateway Config Loading Pipeline
- `hermes-gateway-ops` — # Hermes Gateway Operations & Platform Debugging - `hermes-gateway-ops` — # Hermes Gateway Operations & Platform Debugging
- `hermes-instance-migration` — # Hermes Instance Migration (audit + curated port) - `hermes-instance-migration` — # Hermes Instance Migration (audit + curated port)
- `hermes-instance-update` — # Hermes Instance Update Operations
- `hermes-mattermost-integration` — # Hermes Mattermost Integration - `hermes-mattermost-integration` — # Hermes Mattermost Integration
- `hermes-memory-systems` — # Hermes Agent Memory Systems - `hermes-memory-systems` — # Hermes Agent Memory Systems
- `hermes-multi-agent` — # Hermes Multi-Agent Architecture - `hermes-multi-agent` — # Hermes Multi-Agent Architecture
@@ -238,7 +232,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `domain-driven-design` — # Domain-Driven Design Framework - `domain-driven-design` — # Domain-Driven Design Framework
## Email (17) ## Email (16)
- `agent-email-patterns` — # Agent Email Patterns - `agent-email-patterns` — # Agent Email Patterns
- `agentmail` — # AgentMail — Agent-Owned Email Inboxes - `agentmail` — # AgentMail — Agent-Owned Email Inboxes
@@ -255,7 +249,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `email-inbox-triage` — # Email Inbox Triage - `email-inbox-triage` — # Email Inbox Triage
- `gmail-readonly-fetch` — # Gmail Read-Only Fetch (Large Inbox) - `gmail-readonly-fetch` — # Gmail Read-Only Fetch (Large Inbox)
- `himalaya` — # Himalaya Email CLI - `himalaya` — # Himalaya Email CLI
- `mail-lookup` — # Ad-Hoc Mail Lookup
- `realtime-email-monitor` — # Real-time Email Monitor (IMAP IDLE) - `realtime-email-monitor` — # Real-time Email Monitor (IMAP IDLE)
## Email Himalaya (1) ## Email Himalaya (1)
@@ -388,7 +381,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `pragmatic-programmer` — # The Pragmatic Programmer Framework - `pragmatic-programmer` — # The Pragmatic Programmer Framework
## Productivity (25) ## Productivity (26)
- `airtable` — # Airtable — Bases, Tables & Records - `airtable` — # Airtable — Bases, Tables & Records
- `box` — # Box - `box` — # Box
@@ -398,6 +391,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
- `google-workspace` — # Google Workspace - `google-workspace` — # Google Workspace
- `maps` — # Maps Skill - `maps` — # Maps Skill
- `meeting-action-items` — # Meeting Action Items - `meeting-action-items` — # Meeting Action Items
- `microsoft-office` — # Microsoft Office Skill
- `nano-pdf` — # nano-pdf - `nano-pdf` — # nano-pdf
- `notion` — # Notion - `notion` — # Notion
- `notion-database-dedup` — # Notion Database Dedup - `notion-database-dedup` — # Notion Database Dedup