audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129
This commit is contained in:
@@ -3,8 +3,8 @@ title: Hermes Chaos
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, chaos, ct450, experimental, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -28,7 +28,7 @@ Experimental Hermes instance for testing new configurations, skills, themes, mod
|
||||
## Current State
|
||||
|
||||
- **Status:** Active (experimental)
|
||||
- **Hermes version:** `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 now returns **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim is finally true. `hermes update --check` reports stable **v0.23.0** (`2bd378ed1b8d`) available. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Hermes version:** ⚠ **now `v0.23.0`** — the live install stamp `~/.hermes/hermes-agent/install-stamp.json` inside CT450 reads `baseVersion: 0.23.0`, `displayVersion: 0.23.0`, `distance: 0`, `payload: bootstrap` (probed 2026-11-08 00:4x AWST by the nightly audit; `hermes --version` not re-run in that probe). The rest of this bullet is the 2026-11-07 correction, written when CT450 was still `v0.22.5` (2026.9.24), local `bf0203a794a0` — **the same build as CT460** (updated 2026-10-21). ⚠ **Corrected 2026-11-07:** this line still read *"`v0.22.3` … local `5b147ab036d6`"* while asserting it was *"the same build as CT460"* — it was not. Live `hermes --version` on CT450 then returned **`v0.22.5` … local `bf0203a794a0`**, byte-identical to CT460's own `hermes --version`, so the *same build* claim finally became true. `hermes update --check` then reported stable **v0.23.0** (`2bd378ed1b8d`) available; CT460 took it on 2026-11-07 and CT450's stamp now reads the same version. Python 3.14.9. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||||
- **Mattermost connection failing:** the gateway logs `mattermost connect timed out after 30s, next retry in 300s` every 5 minutes. Host-side condition (Mattermost unreachable), not a CT450 regression — see [[systems/messaging-integrations]].
|
||||
- **Soul:** Chaos — the unhinged raccoon 🦝 (SOUL.md beat-bank variant, updated 2026-08-01)
|
||||
- **Cron jobs:** **25 — 13 enabled / 12 disabled** (verified live in CT450's `jobs.json` **2026-11-07**; unchanged). **9 job names are shared with CT460**; 8 of those are disabled here and enabled on CT460, while `Important-mail monitor` is the reverse (enabled on CT450, disabled on CT460) — both confirmed by a live set-diff of the two `jobs.json` files on 2026-11-07
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Production
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-08-15
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [hermes, production, ct460, agent]
|
||||
sources: [live-probe, config-inspection]
|
||||
@@ -27,7 +27,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
## Current State
|
||||
|
||||
- **Status:** Active
|
||||
- **Hermes version:** `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST; **verified live 2026-10-26**. (`hermes update --check` reports stable **v0.22.5** available)
|
||||
- **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST.
|
||||
- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||
- **Config:** `~/.hermes/config.yaml`
|
||||
|
||||
+43
-1
@@ -12,7 +12,49 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-11-07
|
||||
Updated: 2026-11-08
|
||||
|
||||
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
|
||||
|
||||
- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10,
|
||||
`bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot`
|
||||
stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**.
|
||||
Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55
|
||||
(live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]]
|
||||
- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in
|
||||
*after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health
|
||||
Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance*
|
||||
(`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on
|
||||
`lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00**
|
||||
(`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and
|
||||
`1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with
|
||||
**`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06.
|
||||
- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`,
|
||||
2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not
|
||||
left dirty.
|
||||
- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update.
|
||||
No `browser.*` config key changed. → [[systems/browser-backend]]
|
||||
- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own
|
||||
last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. →
|
||||
[[systems/conduit]]
|
||||
- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web
|
||||
routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. →
|
||||
[[systems/hermes-hq-push]]
|
||||
- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on
|
||||
`…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07
|
||||
execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]]
|
||||
- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into
|
||||
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew
|
||||
**46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the
|
||||
*"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity
|
||||
25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]]
|
||||
- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7
|
||||
`delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. →
|
||||
[[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check
|
||||
found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.
|
||||
|
||||
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)
|
||||
|
||||
|
||||
@@ -105,6 +105,19 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c
|
||||
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
|
||||
scrubbed.
|
||||
|
||||
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
|
||||
|
||||
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
|
||||
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
|
||||
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
|
||||
would still pass.
|
||||
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
|
||||
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
|
||||
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
|
||||
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
|
||||
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
|
||||
executed. Both **Remediation options 2 and 4 above remain open**.
|
||||
|
||||
## See also
|
||||
|
||||
- [[infrastructure/credentials-map]] - where secrets live, by design
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-08
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
|
||||
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
|
||||
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
|
||||
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
|
||||
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
|
||||
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
|
||||
|
||||
| job | id | run window (ledger) | closed |
|
||||
|---|---|---|---|
|
||||
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
|
||||
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
|
||||
|
||||
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
|
||||
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
|
||||
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
|
||||
|
||||
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
|
||||
|
||||
| time | event |
|
||||
|---|---|
|
||||
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
|
||||
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
|
||||
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
|
||||
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
|
||||
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
|
||||
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
|
||||
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
|
||||
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
|
||||
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
|
||||
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
|
||||
| 05:01:41 | gateway back up as `pid 2981474` |
|
||||
| 05:02:22 | both ledger rows finalised `failed` |
|
||||
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
|
||||
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
|
||||
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
|
||||
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
|
||||
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
|
||||
|
||||
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
|
||||
|
||||
| id | job | state | first seen | alerted_at | closed |
|
||||
|---|---|---|---|---|---|
|
||||
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
|
||||
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
|
||||
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
|
||||
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
|
||||
|
||||
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
|
||||
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
|
||||
|
||||
## Root cause and working hypotheses
|
||||
|
||||
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
|
||||
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
|
||||
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
|
||||
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
|
||||
remaining work.
|
||||
|
||||
**Not established — the restart's origin.** The request at ~04:29:52 is:
|
||||
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
|
||||
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
|
||||
step is recorded as **`restarting_gateway` = failed**.
|
||||
|
||||
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
|
||||
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
|
||||
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
|
||||
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
|
||||
does not begin until **05:05:13**, after the restart was already pending.
|
||||
|
||||
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
|
||||
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
|
||||
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
|
||||
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
|
||||
2026-11-07 edits are known to exist only because an in-day session committed them
|
||||
([[systems/scheduled-tasks]]).
|
||||
|
||||
## Impact
|
||||
|
||||
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
|
||||
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
|
||||
tail ([[systems/mnemosyne]]).
|
||||
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
|
||||
`last_status: error`) — no stocktake ran on 2026-11-07.
|
||||
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
|
||||
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
|
||||
|
||||
## Verification evidence
|
||||
|
||||
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
|
||||
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
|
||||
/home/hermes/.hermes/conduit-self-update.json
|
||||
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
|
||||
|
||||
## Open items
|
||||
|
||||
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
|
||||
fix the alerting path that let a `detected` row sit silent.
|
||||
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
|
||||
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
|
||||
@@ -2330,3 +2330,57 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
||||
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
|
||||
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
|
||||
included in this commit so the tree is not left dirty.
|
||||
|
||||
## [2026-11-08] audit - window 2026-11-07
|
||||
|
||||
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
|
||||
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
|
||||
0 `for_failure`.
|
||||
- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
|
||||
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
|
||||
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
|
||||
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
|
||||
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
|
||||
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
|
||||
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
|
||||
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
|
||||
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
|
||||
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13).
|
||||
- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
|
||||
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
|
||||
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52.
|
||||
- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`);
|
||||
the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
|
||||
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
|
||||
`pid 3109529`). CT450's install stamp now also reads **0.23.0**.
|
||||
- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
|
||||
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
|
||||
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
|
||||
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
|
||||
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
|
||||
an auth error.
|
||||
- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
|
||||
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
|
||||
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed.
|
||||
- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
|
||||
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
|
||||
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
|
||||
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
|
||||
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy).
|
||||
- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`.
|
||||
- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
|
||||
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
|
||||
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
|
||||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
|
||||
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
|
||||
(`/home/hermes/wiki`).
|
||||
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
|
||||
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
|
||||
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
|
||||
**2026-11-08 00:38:29 AWST**.
|
||||
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
|
||||
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
|
||||
the upstream error code only.
|
||||
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# Wiki freshness audit — 2026-11-08
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (0)
|
||||
|
||||
|
||||
## FRESH (126)
|
||||
|
||||
- [ 19d] CONTRIBUTING.md
|
||||
- [ 0d] agents/hermes-chaos.md
|
||||
- [ 0d] agents/hermes-production.md
|
||||
- [ 19d] agents/index.md
|
||||
- [ 10d] agents/nanobot.md
|
||||
- [ 19d] agents/open-claw.md
|
||||
- [ 4d] agents/qwenpaw.md
|
||||
- [ 19d] agents/zeroclaw.md
|
||||
- [ 10d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 19d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 19d] comparisons/memory-backends.md
|
||||
- [ 19d] comparisons/model-routers.md
|
||||
- [ 10d] comparisons/steel-vs-camoufox.md
|
||||
- [ 19d] concepts/agent-memory.md
|
||||
- [ 10d] concepts/browser-automation.md
|
||||
- [ 19d] concepts/model-routing.md
|
||||
- [ 19d] concepts/persistent-shells.md
|
||||
- [ 19d] concepts/tool-calling.md
|
||||
- [ 1d] current-state.md
|
||||
- [ 10d] decisions/index.md
|
||||
- [ 19d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 19d] decisions/persistent-shell-enabled.md
|
||||
- [ 10d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 19d] experiments/index.md
|
||||
- [ 15d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 19d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 15d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 1d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 3d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 4d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||
- [ 14d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 19d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 18d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 4d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 10d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 13d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 13d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 10d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 15d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 7d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 2d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 12d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 20d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 12d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 17d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 14d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 13d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 13d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 13d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 11d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 9d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 9d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 9d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 8d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 9d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 8d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 7d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 7d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 7d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 6d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 5d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 4d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 2d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||
- [ 2d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||
- [ 3d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||
- [ 2d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||
- [ 3d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||
- [ 1d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||
- [ 0d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||
- [ 1d] index.md
|
||||
- [ 1d] infrastructure/backups.md
|
||||
- [ 19d] infrastructure/credentials-map.md
|
||||
- [ 1d] infrastructure/docker-services.md
|
||||
- [ 19d] infrastructure/domains-and-tunnels.md
|
||||
- [ 1d] infrastructure/hosts.md
|
||||
- [ 1d] infrastructure/network-map.md
|
||||
- [ 1d] infrastructure/proxmox.md
|
||||
- [ 1d] infrastructure/tailscale.md
|
||||
- [ 4d] queries/retained-research-answers.md
|
||||
- [ 14d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 7d] reports/uber-eats/2026-07.md
|
||||
- [ 7d] reports/uber-eats/2026-08.md
|
||||
- [ 7d] reports/uber-eats/2026-09.md
|
||||
- [ 7d] reports/uber-eats/2026-10.md
|
||||
- [ 7d] reports/uber-eats/index.md
|
||||
- [ 19d] runbooks/backup-wiki.md
|
||||
- [ 19d] runbooks/diagnose-docker.md
|
||||
- [ 19d] runbooks/gateway-resource-resilience.md
|
||||
- [ 1d] runbooks/job-radar.md
|
||||
- [ 19d] runbooks/multiplexer-setup.md
|
||||
- [ 13d] runbooks/provider-health.md
|
||||
- [ 10d] runbooks/recover-docker-service.md
|
||||
- [ 19d] runbooks/restart-browser.md
|
||||
- [ 19d] runbooks/restart-hermes.md
|
||||
- [ 19d] runbooks/restore-openrouter.md
|
||||
- [ 19d] runbooks/test-openrouter.md
|
||||
- [ 19d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 13d] runbooks/update-hermes-safely.md
|
||||
- [ 0d] systems/browser-backend.md
|
||||
- [ 19d] systems/cloudflare-tunnel.md
|
||||
- [ 0d] systems/conduit.md
|
||||
- [ 13d] systems/current-profile.md
|
||||
- [ 19d] systems/guanaco.md
|
||||
- [ 19d] systems/headless-server.md
|
||||
- [ 0d] systems/hermes-agent.md
|
||||
- [ 0d] systems/hermes-hq-push.md
|
||||
- [ 19d] systems/hermes-ui.md
|
||||
- [ 4d] systems/home-assistant.md
|
||||
- [ 9d] systems/homepage-dashboard.md
|
||||
- [ 4d] systems/ignis.md
|
||||
- [ 13d] systems/memory-backend.md
|
||||
- [ 6d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 13d] systems/model-providers.md
|
||||
- [ 4d] systems/nanobot.md
|
||||
- [ 19d] systems/nextcloud.md
|
||||
- [ 19d] systems/obsidian.md
|
||||
- [ 7d] systems/omniroute.md
|
||||
- [ 0d] systems/opencode-go.md
|
||||
- [ 19d] systems/openrouter.md
|
||||
- [ 4d] systems/paseo.md
|
||||
- [ 20d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 19d] systems/terminal-backend.md
|
||||
- [ 19d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 0 stale, 126 fresh, 3 no-date of 129 pages
|
||||
@@ -3,8 +3,8 @@ title: Browser Backend
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [browser, camofox, automation]
|
||||
sources: [live-probe, config, session-history]
|
||||
@@ -16,6 +16,13 @@ sources: [live-probe, config, session-history]
|
||||
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
|
||||
|
||||
## Current Configuration (re-verified 2026-10-13)
|
||||
|
||||
> **2026-11-07/08 — the browser backend now ships as a plugin, not a core feature.** The Hermes
|
||||
> `0.22.5 → 0.23.0` update installed **`camofox` v1.1.0** as an **enabled plugin** at
|
||||
> `~/.hermes/plugins/camofox/` (`plugin.yaml`, `camofox_tools.py`, `camofox_policy.py`,
|
||||
> `camofox_vault.py`, `camofox_cli.py`), confirmed live by `hermes plugins list` on 2026-11-07.
|
||||
> No `browser.*` key was changed in this window — the sections below still describe the running
|
||||
> configuration ([[systems/hermes-agent]]).
|
||||
- **Primary backend:** Camofox (Firefox-based, anti-fingerprint) — `systemd --user` unit `camofox-browser.service` **on the Hermes host itself** (the unit path is `/home/hermes/.config/systemd/user/…`, and the VNC route is named after CT460). The older `CT450` attribution below was not reproducible on 2026-10-13 — see the correction note.
|
||||
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||||
- **REST API:** **`:9377`** — live-verified 2026-10-13 (`node server.js`, `CAMOFOX_PORT=9377`; `LISTEN *:9377`). **The `:9093` figure in earlier revisions of this page is stale** — nothing was listening on 9093 on 2026-10-13.
|
||||
|
||||
+26
-3
@@ -3,9 +3,9 @@ title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-12-08
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
|
||||
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329, session-20260908_031541_49c051]
|
||||
@@ -207,6 +207,29 @@ mode never applies** despite a passing connection test.
|
||||
**4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy.
|
||||
- **Not re-measured:** per-kind event split, health-table freshness, device pairing state.
|
||||
|
||||
## 2026-11-08 00:4x AWST — self-updated again to 0.134.4, and its last step failed
|
||||
|
||||
The plugin updated itself in-window and its own ledger records the final step as **failed**:
|
||||
|
||||
~/.hermes/conduit-self-update.json
|
||||
{"id": "916d72daeb644b06b5ae1359ddfdd992", "step": "failed", "from_version": "0.122.2",
|
||||
"version": "0.134.4", "failed_step": "restarting_gateway",
|
||||
"detail": "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host.",
|
||||
"started_at": 1793996875.92, "finished_at": 1793997152.39, "pid": 176771, "boot_id": "f292ee44712be8b3"}
|
||||
|
||||
- Window: **2026-11-07 04:27:55 → 04:32:32 AWST** (276 s). `from_version` **0.122.2** (the version this page
|
||||
recorded on 2026-11-01) → **0.134.4**; `hermes plugins list` and the plugin tree agree.
|
||||
- `failed_step: restarting_gateway` is literal: the gateway was **not** restarted at 04:32 — `pid 176771`
|
||||
was still the live gateway at 04:59 and only restarted at **05:01:00** (systemd), then again at 05:43 as
|
||||
part of the Hermes 0.22.5 → 0.23.0 update ([[systems/hermes-agent]]).
|
||||
- **Consequence, and the reason this is recorded here:** a restart was requested at ~04:29:52 and held for
|
||||
the full 1800 s *after-turn* window, then drained — which killed two in-flight cron runs
|
||||
([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]). *Working hypothesis, not proven:* the
|
||||
requester was this self-update. Either way, **a plugin under `~/.hermes/plugins/` cannot restart a
|
||||
systemd-managed gateway itself**, and it reported failure rather than retrying.
|
||||
- The plugin remained enabled and functional throughout; no user-facing conduit fault was recorded on
|
||||
2026-11-07.
|
||||
|
||||
## Open items
|
||||
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
|
||||
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
@@ -16,7 +16,7 @@ sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
||||
|
||||
## Current Status
|
||||
- **Version:** `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
|
||||
- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
|
||||
- **Install method:** git
|
||||
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
||||
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
|
||||
|
||||
@@ -3,9 +3,9 @@ title: hermes-hq-push (Hermes HQ iPhone app plugin)
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-12-04
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-12-08
|
||||
confidence: medium
|
||||
tags: [push, notifications, plugins, ios, desktop, device-access]
|
||||
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
|
||||
@@ -15,8 +15,8 @@ sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
|
||||
|
||||
## What it is
|
||||
|
||||
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*)
|
||||
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half
|
||||
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.42**, author *Hermes HQ*)
|
||||
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-07 06:42**, reinstalled 0.9.27 → 0.9.42). It is the host-side half
|
||||
of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal
|
||||
page, and Grok Live voice token minting.
|
||||
|
||||
@@ -93,6 +93,24 @@ No key material is recorded on this page and none appears in the manifest — th
|
||||
- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the
|
||||
verification table above).
|
||||
|
||||
## 2026-11-07 — reinstalled to v0.9.42, routes picked up by a gateway restart
|
||||
|
||||
- The plugin was **reinstalled to `v0.9.42`** in this window (from `v0.9.27`) and the **gateway was
|
||||
restarted at 11:11:44 → 11:11:55 AWST** so the new tool routes were registered (live process then
|
||||
`pid 3109529`). A plugin's routes are only registered at gateway start, so a reinstall without a restart
|
||||
changes nothing on the wire — the restart is the delivery step, not a symptom
|
||||
([[systems/hermes-agent]]).
|
||||
- Same night, the plugin tree's mtime moved from **2026-11-03 11:06** (the mtime this page recorded on
|
||||
2026-11-04) to **2026-11-07 06:42**; `hermes plugins list` on 2026-11-07 reports `hermes-hq-push` **0.9.42**
|
||||
enabled (`plugin.yaml` agrees). Installed from `mrcharlesiv/hermes-hq-plugins`, upstream **`d26850e7`**,
|
||||
verified complete at **60/60 files**. Order of events that made the restart necessary:
|
||||
`hermes-dashboard` started **05:43:04** and `hermes-gateway` **05:37:34** (the 0.22.5 → 0.23.0 update) —
|
||||
*before* the 06:42 reinstall, so neither running process had the new routes. A verification pass at
|
||||
10:00:46 confirmed the plugin files were newer than both service starts, and the single restart followed
|
||||
at 11:11:44 → 11:11:55 (`pid 3109529`).
|
||||
- No app-side fault was recorded on 2026-11-07. Push delivery, device access and Grok Live minting were not
|
||||
re-exercised end-to-end by this audit — the version/route claim is the verified part.
|
||||
|
||||
## Open
|
||||
|
||||
- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk
|
||||
|
||||
+27
-2
@@ -3,8 +3,8 @@ title: OpenCode Go
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [opencode, provider, models, subscription, limits]
|
||||
sources: [https://opencode.ai/docs/go/]
|
||||
@@ -137,6 +137,31 @@ With Go, you pay $10/month and they aim to give you **6x that in usage** ($60).
|
||||
| Qwen3.6 Plus | `qwen3.6-plus` | same | same |
|
||||
| Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
|
||||
|
||||
## 2026-11-08 00:0x-00:5x AWST — the stored credential is expired (401 `api_key_expired`)
|
||||
|
||||
**Status: `active` as a configuration, `dead` as a credential.** Two independent live probes in this window
|
||||
both returned **HTTP 401** on the Go endpoint:
|
||||
|
||||
probe 1 (this audit, 2026-11-08 00:2x): zen_go_http=401
|
||||
{"error":{"type":"authentication_error","code":"api_key_expired",
|
||||
"message":"This API key has expired. Create a new API key in OpenCode Console and
|
||||
replace it in your client or secret store. …"}}
|
||||
— the same 401 for `https://opencode.ai/zen/go/v1/models`, and `~/.local/share/opencode/auth.json`
|
||||
still lists an *OpenCode Go api* entry (the CLI's own store).
|
||||
probe 2 (2026-11-07): provider=opencode-go (fields `key`, `type`) → HTTP 401, body
|
||||
`{"error":{"type":"authentication_error","code":"api_key_expired", …}}`
|
||||
|
||||
- This is **upstream expiry, not a Hermes fault**: the key was issued by the OpenCode console and has run
|
||||
out. Hermes is still configured for the provider (`opencode-go/<model-id>`, per the table above), and
|
||||
router bridges exist (`ollama-opencode-go-deepseek-v4-flash`, `-glm-5*`, `-grok-4_6`, `-kimi-k2_5`, …).
|
||||
- **No 2026-11-07 execution failed on an auth error** — the day's three failures were all the gateway
|
||||
restart drain ([[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]), so nothing in the fleet
|
||||
was actually consuming this provider in-window. The risk is latent: anything pinned to it fails on first
|
||||
use.
|
||||
- **To restore:** mint a new key in the OpenCode console, then replace it in **both** stores — the Hermes
|
||||
provider entry and `~/.local/share/opencode/auth.json` — and re-probe `…/zen/go/v1/models` for a 200.
|
||||
(Keys live in the vault; this page records only their state — [[references/hermes-api-vault]].)
|
||||
|
||||
**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
|
||||
|
||||
**Model list API:** `https://opencode.ai/zen/go/v1/models`
|
||||
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2026-11-08
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -1155,3 +1155,82 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
|
||||
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
|
||||
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed.
|
||||
|
||||
## Re-verified 2026-11-08 (window 2026-11-07, AWST; live reads 2026-11-08 00:0x-00:5x)
|
||||
|
||||
### The day
|
||||
|
||||
- **2026-11-07 settled 163 ledger rows — 159 `completed` / 3 `failed` / 1 `running`** (the *running* row is
|
||||
this audit, `cbfe434a3b5e`, fired 23:59:58). Same fleet as the previous five audits: **29 jobs, 26 enabled**.
|
||||
- **Deliveries: 7 rows / 7 `delivered` / 0 `for_failure`** (10 on each of 11-05 and 11-06 — no delivery fault
|
||||
to explain).
|
||||
|
||||
### The window's failure: a restart drain orphaned two in-flight runs
|
||||
|
||||
- Two of the three `failed` rows are **one event**. `ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 →
|
||||
05:02:22.870) and `1a8e90ea2271` (*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174) both closed with
|
||||
`RuntimeError: Cron job '…' lost its durable fire claim ownership`. The drain happened at **05:00:43/47**;
|
||||
systemd restarted the gateway at **05:01:00** (`Scheduled restart job, restart counter is at 3.`) after a
|
||||
requested restart sat in *after-turn* mode for the full **1800 s** (`#77184`).
|
||||
- The restart was requested at **~04:29:52** — 2 s before both *restart-safe cron workers* exited status 1
|
||||
(*"after adopting this execution, without writing a durable terminal state; whether side effects ran is
|
||||
unknown"*, logged 04:29:54) — i.e. **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`,
|
||||
04:27:55 → 04:32:32, its own ledger recording `restarting_gateway` = **failed**). Working hypothesis, not
|
||||
proven: the plugin asked for the restart and could not perform it. The 0.22.5 → 0.23.0 updater is excluded
|
||||
(its receipt starts 05:05:13). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- The third `failed` row is **the watchdog working**: `96eb95ce12f2` (05:02:41.105 → 05:02:59.640) exited 1
|
||||
having caught the health monitor's `RuntimeError` traceback in `agent.log` — it reported the event ~19 s
|
||||
after it was written.
|
||||
|
||||
### Still open
|
||||
|
||||
- ⚠ **Unclosed incident rows 0 → 2, and neither alerted.** Both belong to the stocktake job and sit in state
|
||||
`detected` with **`alerted_at: null`**: `1a8e90_2f370ef764f5` (first seen 2026-11-07T05:02:21.573) and
|
||||
`1a8e90_628d914f1f97` (2026-11-07T04:29:54.815). The health monitor's two rows were closed 15:43:17 and
|
||||
resolved 17:08:52 the same day. This is the *red row, no alert* shape the 11-06 audit recorded for Job
|
||||
Radar, now reproduced on a different job.
|
||||
- ⚠ **Non-clean `last_status` 1 → 2:** `1a8e90ea2271` is now `error` with **`failure_streak: 2`** — the 03:00
|
||||
stocktake slot has failed on consecutive days (11-06 and 11-07) and again left its stocktake edits
|
||||
uncommitted; the 11-07 in-day entry folded those edits into its own commit so the tree was not left dirty.
|
||||
The other non-clean row is unchanged: `066bb5191280` (*Uber Eats Monthly Spend Report*), `delivery_failed`,
|
||||
streak 0, *"configured thread_id 34795 … was not found"*.
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] stays **OPEN** (untouched in-window).
|
||||
- **Job Radar ran clean on 11-07** — no slot failed, no claim conflict; the 11-06 concurrency page is
|
||||
unchanged.
|
||||
|
||||
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
|
||||
|
||||
- *Mnemosyne Health Monitor* `ea79a2107e36` — 5 h 01 m before the drain killed it (its two previous slots ran
|
||||
7 h 47 m 56 s and 12 h 13 m 54 s without losing the claim).
|
||||
- *wiki-stocktake-maintenance* `1a8e90ea2271` — 1 h 59 m 51 s before the drain killed it.
|
||||
|
||||
### Skill inventory movement (in-window — 2026-11-07 07:43 AWST)
|
||||
|
||||
- The stocktake's skills-index regeneration now reads **299 skills / 67 categories**, down from **302 / 68**,
|
||||
because **four skills were moved into `~/.hermes/skills/.archive/`** at **07:43**: `cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`. The parked archive grew **46 → 50**,
|
||||
and the `Cron Model Repair` category vanished with its only member.
|
||||
- ⚠ **No active skill references any of the four** — a grep of every `SKILL.md` outside `.archive/` returns
|
||||
no absorbing skill, so if this was a curator consolidation the *"absorbed into X"* pointer is missing.
|
||||
Their content survives only in the parked copies.
|
||||
- One skill **appeared**: `microsoft-office` (Productivity 25 → 26), part of a **10-file skill refresh at
|
||||
05:36** that lines up with the 0.22.5 → 0.23.0 update's own skill materialisation (the others were
|
||||
`computer-use`, `claude-design`, `popular-web-designs`, `document-to-action-items`, `meeting-action-items`,
|
||||
`product-price-monitor`, `weekly-review-planning`, `grounded-citations`, `dogfood`).
|
||||
- Filesystem vs. page: **300 `SKILL.md` on disk vs. 299 listed** — the generator de-dups one nested
|
||||
duplicate, so **299** is the unique figure (behaviour unchanged).
|
||||
|
||||
### Freshness / files touched
|
||||
|
||||
- **Freshness re-run:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261108.txt 21`
|
||||
→ **0 stale / 126 fresh / 3 no-date of 129 pages** (report: `raw/stocktake/freshness-20261108.txt`).
|
||||
Nothing has crossed the 21-day line; the oldest pages are `CONTRIBUTING.md` and `agents/index.md` at 19 d.
|
||||
- **Pages touched by this audit:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
|
||||
[[systems/hermes-agent]] (version → 0.23.0, restart record),
|
||||
[[agents/hermes-production]] (version, 11-07 ledger), [[agents/hermes-chaos]] (CT450 stamp now 0.23.0),
|
||||
[[systems/browser-backend]] (camofox is now a plugin), [[systems/conduit]] (self-update 0.134.4, failed
|
||||
`restarting_gateway`), [[systems/hermes-hq-push]] (0.9.27 → 0.9.42), [[systems/opencode-go]] (stored key
|
||||
expired), [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (redaction-off standing
|
||||
condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both
|
||||
- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level.
|
||||
|
||||
+8
-14
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
@@ -12,9 +12,9 @@ sources: []
|
||||
|
||||
# Installed Skills
|
||||
|
||||
**Total installed: 302 skills** in `~/.hermes/skills/`.
|
||||
**Total installed: 299 skills** in `~/.hermes/skills/`.
|
||||
|
||||
- **46 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||
- **50 archived skills** are parked under `~/.hermes/skills/.archive/` and are excluded from this count — the loader does not expose them.
|
||||
- The loader's own catalogue (`hermes skills list`, e.g. *34 hub-installed, 52 builtin, N local — N enabled*) is the authoritative loadable figure; this page is a filesystem inventory.
|
||||
|
||||
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
@@ -90,10 +90,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `cron-digest-formatting` — # Cron Digest Formatting (Telegram)
|
||||
|
||||
## Cron Model Repair (1)
|
||||
|
||||
- `cron-model-repair` — # Cron Model-Pin Repair
|
||||
|
||||
## Data Science (1)
|
||||
|
||||
- `jupyter-live-kernel` — # Jupyter Live Kernel (hamelnb)
|
||||
@@ -102,7 +98,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
||||
|
||||
## Devops (109)
|
||||
## Devops (107)
|
||||
|
||||
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
||||
- `agent-extension-audit` — # Agent Extension Audit
|
||||
@@ -127,7 +123,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `docker-management` — # Docker Management
|
||||
- `external-auth-script-integration` — # External Auth Script Integration
|
||||
- `external-skill-integration` — # External Skill Integration
|
||||
- `fleet-wide-failure-triage` — # Fleet-Wide Failure Triage
|
||||
- `free-web-tools` — # Free Web Search & Extract
|
||||
- `freshrss-intelligence-filter` — # FreshRSS Intelligence Filter
|
||||
- `gcp-vertex-ai-provider` — # GCP Vertex AI Provider
|
||||
@@ -143,7 +138,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `hermes-gateway-config-loading` — # Hermes Gateway Config Loading Pipeline
|
||||
- `hermes-gateway-ops` — # Hermes Gateway Operations & Platform Debugging
|
||||
- `hermes-instance-migration` — # Hermes Instance Migration (audit + curated port)
|
||||
- `hermes-instance-update` — # Hermes Instance Update Operations
|
||||
- `hermes-mattermost-integration` — # Hermes Mattermost Integration
|
||||
- `hermes-memory-systems` — # Hermes Agent Memory Systems
|
||||
- `hermes-multi-agent` — # Hermes Multi-Agent Architecture
|
||||
@@ -238,7 +232,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `domain-driven-design` — # Domain-Driven Design Framework
|
||||
|
||||
## Email (17)
|
||||
## Email (16)
|
||||
|
||||
- `agent-email-patterns` — # Agent Email Patterns
|
||||
- `agentmail` — # AgentMail — Agent-Owned Email Inboxes
|
||||
@@ -255,7 +249,6 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `email-inbox-triage` — # Email Inbox Triage
|
||||
- `gmail-readonly-fetch` — # Gmail Read-Only Fetch (Large Inbox)
|
||||
- `himalaya` — # Himalaya Email CLI
|
||||
- `mail-lookup` — # Ad-Hoc Mail Lookup
|
||||
- `realtime-email-monitor` — # Real-time Email Monitor (IMAP IDLE)
|
||||
|
||||
## Email Himalaya (1)
|
||||
@@ -388,7 +381,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `pragmatic-programmer` — # The Pragmatic Programmer Framework
|
||||
|
||||
## Productivity (25)
|
||||
## Productivity (26)
|
||||
|
||||
- `airtable` — # Airtable — Bases, Tables & Records
|
||||
- `box` — # Box
|
||||
@@ -398,6 +391,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `google-workspace` — # Google Workspace
|
||||
- `maps` — # Maps Skill
|
||||
- `meeting-action-items` — # Meeting Action Items
|
||||
- `microsoft-office` — # Microsoft Office Skill
|
||||
- `nano-pdf` — # nano-pdf
|
||||
- `notion` — # Notion
|
||||
- `notion-database-dedup` — # Notion Database Dedup
|
||||
|
||||
Reference in New Issue
Block a user