From 785fbc40b20d469e31fc8cf0ac92e558e641e5c5 Mon Sep 17 00:00:00 2001 From: Hermes Wiki Audit Date: Wed, 4 Nov 2026 00:16:23 +0800 Subject: [PATCH] =?UTF-8?q?wiki:=20nightly=20audit=202026-11-03=20window?= =?UTF-8?q?=20=E2=80=94=20new=20Job=20Radar=20stream-timeout=20incident,?= =?UTF-8?q?=20hermes-hq-push=20system=20page;=20update=2011-02=20recurrenc?= =?UTF-8?q?e,=20scheduled-tasks,=20index,=20log?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...b-radar-stage1-iteration-cap-recurrence.md | 39 ++++- ...6-11-03-job-radar-stream-active-timeout.md | 118 +++++++++++++++ index.md | 8 +- log.md | 53 +++++++ raw/stocktake/freshness-20261103.txt | 134 ++++++++++++++++++ systems/hermes-hq-push.md | 104 ++++++++++++++ systems/scheduled-tasks.md | 87 +++++++++++- systems/skills-index.md | 4 +- 8 files changed, 534 insertions(+), 13 deletions(-) create mode 100644 incidents/2026-11-03-job-radar-stream-active-timeout.md create mode 100644 raw/stocktake/freshness-20261103.txt create mode 100644 systems/hermes-hq-push.md diff --git a/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md b/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md index 6e7774a..645700d 100644 --- a/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +++ b/incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md @@ -3,11 +3,11 @@ title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete type: incident status: active created: 2026-11-03 -updated: 2026-11-03 -verified_on: 2026-11-03 -review_after: 2026-11-10 +updated: 2026-11-04 +verified_on: 2026-11-04 +review_after: 2026-11-17 confidence: high -tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox] +tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history] --- @@ -72,9 +72,11 @@ gap is that Stage 1 cannot finish inside its budget. | 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage | | 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff | | **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** | +| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered | Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write -stage never reached. +stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a +different fault, and this time the ledger did go red; see the 2026-11-04 update below.** ## The "green row, no artefact" anti-pattern @@ -98,6 +100,32 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for `cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from `cron_incidents` is not the same as success — here it is the *symptom*. +## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs + +**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the +turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995 +(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` / +`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`, +**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**. +→ [[incidents/2026-11-03-job-radar-stream-active-timeout]] + +**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.** +When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself +kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02: +nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to +Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01 +onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a +stream active timeout."* + +**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run +dies — the iteration cap is the specific case that still escapes as `completed`. + +**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined +silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and +`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and +`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both +unrecovered**, two days running. + ## Open questions 1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in @@ -112,6 +140,7 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for ## Related +- [[incidents/2026-11-03-job-radar-stream-active-timeout]] - [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] - [[incidents/2026-10-27-job-radar-budget-exhausted]] - [[systems/scheduled-tasks]] diff --git a/incidents/2026-11-03-job-radar-stream-active-timeout.md b/incidents/2026-11-03-job-radar-stream-active-timeout.md new file mode 100644 index 0000000..4b708fb --- /dev/null +++ b/incidents/2026-11-03-job-radar-stream-active-timeout.md @@ -0,0 +1,118 @@ +--- +title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff +type: incident +status: open +created: 2026-11-04 +updated: 2026-11-04 +verified_on: 2026-11-04 +review_after: 2026-11-11 +confidence: high +tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history] +--- + +# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff + +## Headline + +**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the +ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1 +(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and +closed: + +| Field | Value | +|---|---| +| `status` | **`failed`** | +| `error` | **`RuntimeError: stream active timeout`** | +| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) | +| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open | + +No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both +Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal +signature** — not the iteration cap that ended 10-27, 10-28 and 11-02. + +## Timeline (all `+08:00`) + +| Time | Event | +|---|---| +| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) | +| 07:01:17 | run session `local-7180cef7…` first message | +| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run | +| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` | +| 12:23:28 | **last tool result of the run** (a `terminal` call) | +| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* | +| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) | +| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens | +| 13:28:51.741724 | **failure notice delivered** | +| 13:28:52.305995 | execution closes `failed` | +| 13:28:55.141618 | incident `alerted` | +| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline | +| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* | + +## The 65-minute silence is the new information + +The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until +**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**. + +Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've +reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This +session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout, +with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never +executed. + +**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the +"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`, +`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the +provider's socket state during the silent window. + +## Why the ledger went red this time + +| Date | `executions.status` | `cron_incidents` row | Notice to Anthony | +|---|---|---|---| +| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report | +| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes | +| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** | +| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** | + +Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and +a delivered notice; the cap produces a green row and silence. That is the distinction +[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed. + +## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost + +| Slot | Job | Ran | Result | +|---|---|---|---| +| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` | +| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` | + +`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` — +nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's +discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success +remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**, +`missing_cover_letter_count 0`). + +## Context churn (observation, cause not established) + +**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five +compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn +contributes to the stream stall is **not established** — this is recorded because both long failures share +the pattern, not because the link is proven. + +## Open questions + +1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die + silently long before the harness noticed? +2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily. +3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be + re-slotted, or Stage 2 made to wait on a handoff? +4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30. +5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the + day's discovery to the next slot? + +## Related + +- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] +- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] +- [[incidents/2026-10-27-job-radar-budget-exhausted]] +- [[runbooks/job-radar]] +- [[systems/scheduled-tasks]] diff --git a/index.md b/index.md index 0511adc..8454911 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-03 -verified_on: 2026-11-03 +updated: 2026-11-04 +verified_on: 2026-11-04 confidence: high tags: [index, infrastructure, systems] --- @@ -13,6 +13,8 @@ tags: [index, infrastructure, systems] > Operational knowledge for Anthony's agents, infrastructure, > model routing and automation. Last updated: 2026-11-03 (nightly audit — the 11-02 fleet settled **177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`**, the **first fully clean ledger day in five**, and **both 11-01 `unknown` rows resolved in-window**: the Sunday 18:00 **Weekly Review reconciled at 01:27:22** after **7 h 26 m 43 s** and **delivered**, so *"the lost review" was never lost* — incident `79f9bb_9c443b602eec` auto-closed **01:27:21.606691** and its page is now **`resolved`** — while `ecc1fc13655a`'s 11-01 ghost `error` cleared on its 12:00:33 slot; **unclosed incident rows 1 → 0** (the first zero in the period), non-clean `last_status` **2 → 1**, deliveries **9 rows / 9 delivered**; ⚠ **but two ~9-hour runs failed invisibly.** The **Job Radar Stage 1** (`dca8482e4f76`, 07:00:33 → 16:01:52, **9 h 01 m 19 s**) hit the **agent iteration cap before writing its handoff**, so **both Stage 2 slots exited `[SILENT]` and nothing reached Notion** — and it **raised no `cron_incidents` row at all**, where the same class raised incidents on 10-27 and 10-28, because its terminal state flipped to `completed` with `error: NULL`; the **03:00 stocktake** (`1a8e90ea2271`) likewise ran **9 h 43 m 29 s**, **delivered no report** and left its output uncommitted in the working tree (committed by this audit).) > +> **2026-11-04 (nightly audit) — the ledger raised its own red row:** the **11-03 fleet settled **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) — **unclosed incident rows 0 → 1**, non-clean `last_status` **1 → 2**, deliveries **7 rows / 7 `delivered`**. The single failure is the **Job Radar Stage 1** (`dca8482e4f76`), which ran **07:00:33 → 13:28:52 (6 h 28 m 18 s)** and died on **`RuntimeError: stream active timeout`** — a **new terminal signature**, not the iteration cap — after going **silent for 65 m 21 s**; its incident `dca848_dfd4bea84f2a` (`timeout`) is **`alerted` and open** and the notice **was delivered** (13:28:51.741724) → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. That answers the 11-02 page's open question: **output is suppressed when the run ends `completed`, delivered when it ends `failed`** — which is what let 11-02 vanish and 11-03 announce itself. Both Stage 2 slots declined correctly, so **2026-11-02's and 2026-11-03's candidate sets are both unrecovered**. ⚠ **The 03:00 stocktake overran for a second day** (`1a8e90ea2271`, 03:00:30 → 10:41:13 = **7 h 40 m 43 s**, `suppressed`, nothing delivered, artefacts uncommitted again). **New page:** [[systems/hermes-hq-push]] (v0.9.27, enabled; manifest says it **replaces `dispatch-push`**, and no trace of the old plugin survives). +> > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > Guest IDs collide between the two hosts. @@ -46,6 +48,7 @@ tags: [index, infrastructure, systems] - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.122.2` (HEAD `9eac857`, 2026-10-31 12:04 — *"Every profile that turns Conduit on runs the main install (#482)"*; **+148 commits** on since the `0.69.2` / `942ddb8` baseline, **working tree clean**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` is now pid 176771, started 2026-10-31 19:35:02** (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** ✅ **That patch is gone** — the tree is clean and **`MEMORY_TIMEOUT_S` ships upstream** in `dashboard/api/memory.py`. ⚠ **New fault: "local" voice mode never applies** — the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` reads **`llm_proxy.requests: 0`**, realtime socket rejects `session.update`), while the app's "test connection" checks **socket reachability only** → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. The call store was re-shaped: **`history` is empty** and `last_request_at` is `null`, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. +- [[systems/hermes-hq-push]] — **Hermes HQ iPhone app plugin** (standalone, **v0.9.27**, `kind: standalone`, author *Hermes HQ*; tree mtime **2026-11-03 11:06**). `hermes plugins list` → **enabled** (state `user`); `config.yaml` `plugins.enabled` line **7556**. Host side of the Hermes HQ app: Apple push (finished turns, lock-screen approvals, "waiting for a password/login/code" alerts, **live Lock Screen cards** sealed to the phone), phone **device access** (contacts, location, reminders, alarms, photos, Health, motion, music), the desktop-app **Hermes HQ theme**, the app's **Terminal page**, fast **Sessions list** on a busy host, cross-posting answers back to Telegram/Discord, a **restart guard**, and **Grok Live** voice-token minting (`XAI_API_KEY` or subscription sign-in; bearer never reaches the phone). Tools `device_contacts_search` / `device_request_result`; 7 hooks. Its manifest states it **replaces `dispatch-push`** — and the replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory remains. **Different stack** from the Cadu/Vory bridge on [[systems/push-notifications]]. ⚠ Its `pre_tool_call` guard also **blocks any tool call whose arguments contain the literal text of a launchd one-off-restart command** — describe it, never quote it. - [[systems/scheduled-tasks]] — **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. @@ -68,6 +71,7 @@ tags: [index, infrastructure, systems] - [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] — **Third iteration-cap in seven days, and the first that raised no incident at all.** Job Radar Stage 1 (`dca8482e4f76`) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** on 2026-11-02, hit the **agent iteration cap before the handoff was written**, and closed **`status: completed` / `error: NULL` / `delivery_outcome: suppressed`**. Both Stage 2 slots declined correctly (`92665337…` 11:30 and the `05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion, no cover letters were generated, and the day's 164 deduped candidates were never persisted** — a state a `status`-only health check reports as green. Contrast 10-27 (`dca848_e93a538ade6b`) and 10-28 (`dca848_0a2f03b5952b`), where the identical cap **did** open rows whose `error` text is the run's own INCOMPLETE report: on 11-02 the ledger **lost the red row it used to produce**. Discovery itself worked — SEEK **51**, LinkedIn **60 IDs**, Indeed live, WA Government **35** (197 raw / **164 new** after three-level dedup) — and a **URL-canonicalisation bug was found and fixed mid-run** (`au.seek.com` vs `au.seek.com.au` had been dropping SEEK *and* Indeed entries from the consolidated list). → [[systems/scheduled-tasks]] - [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] — **The Sunday 18:00 Weekly Review did not run on 2026-11-01 and nothing was delivered:** its execution closed `unknown` at 18:05:13 with *"Cron execution unverified … Gateway connection failed; no local fallback"*, its incident row is **still `alerted`**, the job sits at `error` / `failure_streak: 1`, and `deliveries.db` shows **no row after 11:01:12** for the rest of the day. Its batch-mate `b63ed5817212` took the identical error and self-healed in 68 s. Same page records the **inverse silent failure**: `ecc1fc13655a` reads `error` with `last_run 11-01T12:01:00` while owning **no execution row and no incident row at all**. ✅ **RESOLVED 2026-11-02T01:27:21 — the review was never lost.** The *same* execution row (`be6ec11a8e68428084af9ab18fb148db`) closed **`completed` / `delivered` at 01:27:22** after **7 h 26 m 43 s**, the delivery lands at **01:27:21.802246** and incident `79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** — so the 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**, not a verdict (page now `resolved`). Batch-mate `b63ed5` closed in 68 s and `79f9bb` in 7 h 26 m, both `completed`; **an `unknown` row on a long job is a timing artefact, not a failure**. `ecc1fc13655a`'s ghost `error` cleared too (12:00:33 → `completed` / `delivered` 12:29:26), taking **non-clean `last_status` 2 → 1**. - [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] — **Three restart-safe cron workers killed by `SIGTERM` (`-15`) after adopting their claim**, so the scheduler recorded *"whether side effects ran is unknown"*: `85edf6772bca` @07:30:46, `96eb95ce12f2` + `0e753e9720fd` @09:02:18. **A fourth variant of the orphan class** — and the first with **no gateway restart, reload or drain anywhere in the window** (PID `175269` up since 10-31 19:24:40; relay counter monotonic through both signals), so it cannot be attributed to reload behaviour. Likely trigger: a **30 s worker-ack window** smaller than this host's cold start (**83 plugins / 69 enabled** discovered in 70 s). All three rows now `resolved`; one execution left `unknown`. +- [[incidents/2026-11-03-job-radar-stream-active-timeout]] — **The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the ledger went red.** `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** on 2026-11-03 and closed **`failed` / `RuntimeError: stream active timeout`**, with the **failure notice delivered** (13:28:51.741724) and incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`, **`alerted`**, open) — **unclosed rows 0 → 1**. The run's activity stops at **12:23:28**; the harness does not declare the turn dead until **13:28:49** — a **65 m 21 s silence** with no tool call, no model turn and nothing in the ledger, and **no "maximum number of tool-calling iterations" message anywhere** (this is a provider-stream timeout, not the cap that ended 10-27 / 10-28 / 11-02). It compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) in 363 messages. **Both Stage 2 slots declined correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) because no handoff was written, and `radar-ref/handoff/` still holds only 10-27 / 10-29 / 10-30 — so **11-02's 164 deduped candidates and 11-03's discovery are both unrecovered**. Pulse carried it to Anthony in the desktop session. **Answers the 11-02 page's open question 3:** suppression tracks the terminal state — `completed` → nothing delivered, `failed` → notice delivered. → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]], [[systems/scheduled-tasks]] - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. diff --git a/log.md b/log.md index 64cc16a..cc3fbb0 100644 --- a/log.md +++ b/log.md @@ -1907,3 +1907,56 @@ recorded wrong.** - **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job id, PID, thread id and timestamp above is a non-sensitive operational identifier. + + +## [2026-11-04] audit — window 2026-11-03 + +**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` / +1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since +10-31**. + +**New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1 +(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on +**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35, +08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at +**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident +`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still +open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse +carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream +active timeout"*). + +**Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the +recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery +tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended +`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean +state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`. +Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.** + +**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16, +14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30. + +**New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author +*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`; +`config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the +replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory +remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched, +still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call +whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was +blocked and had to be rephrased. + +**Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:** +`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows, +7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39). + +⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 = +**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts +uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this +audit committed them. + +**Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` +`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the +2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02). + +**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no +decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an +embedded credential was seen during verification and is not reproduced here ([REDACTED]). diff --git a/raw/stocktake/freshness-20261103.txt b/raw/stocktake/freshness-20261103.txt new file mode 100644 index 0000000..728ecba --- /dev/null +++ b/raw/stocktake/freshness-20261103.txt @@ -0,0 +1,134 @@ +# Wiki freshness audit — 2026-11-03 +Threshold: >21 days since `updated` + +## STALE (8) + +- [ 23d] agents/qwenpaw.md (updated 2026-10-11) +- [ 23d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11) +- [ 23d] systems/ignis.md (updated 2026-10-11) +- [ 23d] systems/paseo.md (updated 2026-10-11) +- [ 22d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12) +- [ 22d] queries/retained-research-answers.md (updated 2026-10-12) +- [ 22d] systems/home-assistant.md (updated 2026-10-12) +- [ 22d] systems/nanobot.md (updated 2026-10-12) + +## FRESH (110) + +- [ 14d] CONTRIBUTING.md +- [ 8d] agents/hermes-chaos.md +- [ 8d] agents/hermes-production.md +- [ 14d] agents/index.md +- [ 5d] agents/nanobot.md +- [ 14d] agents/open-claw.md +- [ 14d] agents/zeroclaw.md +- [ 5d] comparisons/camoufox-vs-chrome.md +- [ 14d] comparisons/hermes-vs-nanobot.md +- [ 14d] comparisons/memory-backends.md +- [ 14d] comparisons/model-routers.md +- [ 5d] comparisons/steel-vs-camoufox.md +- [ 14d] concepts/agent-memory.md +- [ 5d] concepts/browser-automation.md +- [ 14d] concepts/model-routing.md +- [ 14d] concepts/persistent-shells.md +- [ 14d] concepts/tool-calling.md +- [ 3d] current-state.md +- [ 5d] decisions/index.md +- [ 14d] decisions/keep-openrouter-as-fallback.md +- [ 14d] decisions/persistent-shell-enabled.md +- [ 5d] decisions/rollback-steel-to-camofox.md +- [ 14d] experiments/index.md +- [ 10d] incidents/2026-07-29-mnemosyne-relay.md +- [ 14d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 10d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 11d] incidents/2026-08-15-skills-index-drift.md +- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 14d] incidents/2026-08-29-gmail-read-state.md +- [ 13d] incidents/2026-09-02-sftpgo-outage.md +- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 8d] incidents/2026-10-03-ct450-hermes-update.md +- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 10d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 2d] incidents/2026-10-09-memory-recall-relevance.md +- [ 18d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 7d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 15d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 7d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 12d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 9d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 8d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 8d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 6d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 4d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 4d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 4d] incidents/2026-10-29-conduit-voice-silence.md +- [ 3d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 4d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 3d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 2d] incidents/2026-10-31-asana-access-failures.md +- [ 2d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 2d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 1d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 0d] index.md +- [ 12d] infrastructure/backups.md +- [ 14d] infrastructure/credentials-map.md +- [ 2d] infrastructure/docker-services.md +- [ 14d] infrastructure/domains-and-tunnels.md +- [ 3d] infrastructure/hosts.md +- [ 20d] infrastructure/network-map.md +- [ 2d] infrastructure/proxmox.md +- [ 2d] infrastructure/tailscale.md +- [ 9d] reports/communities-wa-social-media-audit-2026.md +- [ 2d] reports/uber-eats/2026-07.md +- [ 2d] reports/uber-eats/2026-08.md +- [ 2d] reports/uber-eats/2026-09.md +- [ 2d] reports/uber-eats/2026-10.md +- [ 2d] reports/uber-eats/index.md +- [ 14d] runbooks/backup-wiki.md +- [ 14d] runbooks/diagnose-docker.md +- [ 14d] runbooks/gateway-resource-resilience.md +- [ 4d] runbooks/job-radar.md +- [ 14d] runbooks/multiplexer-setup.md +- [ 8d] runbooks/provider-health.md +- [ 5d] runbooks/recover-docker-service.md +- [ 14d] runbooks/restart-browser.md +- [ 14d] runbooks/restart-hermes.md +- [ 14d] runbooks/restore-openrouter.md +- [ 14d] runbooks/test-openrouter.md +- [ 14d] runbooks/troubleshoot-tool-loop.md +- [ 8d] runbooks/update-hermes-safely.md +- [ 0d] systems/browser-backend.md +- [ 14d] systems/cloudflare-tunnel.md +- [ 2d] systems/conduit.md +- [ 8d] systems/current-profile.md +- [ 14d] systems/guanaco.md +- [ 14d] systems/headless-server.md +- [ 4d] systems/hermes-agent.md +- [ 14d] systems/hermes-ui.md +- [ 4d] systems/homepage-dashboard.md +- [ 8d] systems/memory-backend.md +- [ 1d] systems/messaging-integrations.md +- [ 2d] systems/mnemosyne.md +- [ 8d] systems/model-providers.md +- [ 14d] systems/nextcloud.md +- [ 14d] systems/obsidian.md +- [ 2d] systems/omniroute.md +- [ 14d] systems/opencode-go.md +- [ 14d] systems/openrouter.md +- [ 15d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 14d] systems/terminal-backend.md +- [ 14d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 8 stale, 110 fresh, 3 no-date of 121 pages \ No newline at end of file diff --git a/systems/hermes-hq-push.md b/systems/hermes-hq-push.md new file mode 100644 index 0000000..df1f969 --- /dev/null +++ b/systems/hermes-hq-push.md @@ -0,0 +1,104 @@ +--- +title: hermes-hq-push (Hermes HQ iPhone app plugin) +type: system +status: active +created: 2026-11-04 +updated: 2026-11-04 +verified_on: 2026-11-04 +review_after: 2026-12-04 +confidence: medium +tags: [push, notifications, plugins, ios, desktop, device-access] +sources: [live-hermes-plugins-list, live-plugin-manifest, live-config] +--- + +# hermes-hq-push (Hermes HQ iPhone app plugin) + +## What it is + +`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*) +living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half +of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal +page, and Grok Live voice token minting. + +Its own manifest states it **replaces `dispatch-push`**, and that *"its registered phones carry over."* + +## Verified state (2026-11-04) + +| Check | Result | +|---|---| +| `hermes plugins list` | `hermes-hq-push` — **enabled**, `0.9.27`, state `user` | +| `config.yaml` → `plugins.enabled` | line **7556**: `- hermes-hq-push` | +| `~/.hermes/plugins/dispatch-push/` | **absent** — and no `dispatch-push` reference survives in `config.yaml` | +| live tree | `~/.hermes/plugins/hermes-hq-push/` (module files + `dashboard/` subdir + its own test suite) | +| README | none — the manifest `description` is the only prose | + +The replacement is therefore **clean**: nothing references the retired plugin, and the new one is explicitly +enabled. Enablement and version are config-verified; individual hook firing was not separately captured. + +## Surfaces (grouped from the plugin manifest) + +**Push / approvals** +- Finished turns — who replied and what they said. +- Approvals with the command shown, **Approve / Deny from the lock screen**. +- A bot waiting for a password, login, code or answer — the site or the question, **never the value**. +- **Live Lock Screen cards:** every step the bot takes (e.g. *Running npm test…*, the two before it, how + many) is pushed to the card, *sealed so only the phone can read it*; the card says **Done** when the turn + ends. +- Tells the app which scheduled jobs a bot is still working on. + +**Device access** (app *Settings › Device Access*) — contacts lookup, location, reminders and events, alarms +and timers, photos, Health, motion, music, and leaving an email or text for Anthony to send. This is what +backs the Conduit-style phone sharing on the Hermes HQ app. + +**Desktop app** — adds the **Hermes HQ theme**; opens a shell on the host for the app's Terminal page (the +login shell, as an SSH login would; `plugin-data`'s `terminal.off` disables it); lets a Nous sign-in delete, +archive and rename the chats it created (Hermes refuses them otherwise) and lets the desktop app open chats +the app made (Hermes answers `not_found` otherwise); keeps a bot's **Sessions list fast** on a host that has +run many scheduled jobs; lets the app send into a chat the desktop app has open; page previews; and keeps +the app's performance log on this computer at **`~/.hermes/hq-logs`**. + +**Voice** — mints the app's **Grok Live** voice session a short-lived xAI token (subscription sign-in or +`XAI_API_KEY`; the bearer itself never reaches the phone). + +**Restart guard** — stops a bot's one-off macOS launchd restart job from looping (those jobs are KeepAlive, +so a one-time restart re-runs forever and locks the app out), and tells a restart script which in-app chats +are mid-reply so a restart can wait. + +**Cross-posting** — on request, posts the bot's answer back to the Telegram or Discord chat a conversation +came from. + +## Interface + +- **Tools:** `device_contacts_search`, `device_request_result`. +- **Hooks:** `pre_approval_request`, `post_approval_response`, `pre_tool_call`, `on_human_input_request`, + `on_human_input_resolved`, `pre_llm_call`, `on_session_end`. + +## Configuration + +All environment variables are optional. `DISPATCH_*` spellings of the APNs/relay names still work. + +| Variable | Purpose | +|---|---| +| `HERMES_HQ_APNS_KEY_PATH`, `HERMES_HQ_APNS_KEY_ID`, `HERMES_HQ_APNS_TEAM_ID` | **Direct APNs mode** — path to the app's `.p8` auth key, its Key ID, and the Apple Developer Team ID that owns the app. *Only the app's developer has these.* | +| `HERMES_HQ_PUSH_RELAY_URL` | Another push relay's HTTPS URL, or `off` to send nothing. **Unset = alerts go through the public Hermes HQ relay, sealed so it cannot read them.** | +| `HERMES_HQ_PUSH_RELAY_KEY` | Only for a private relay that asks for a key. | + +No key material is recorded on this page and none appears in the manifest — the relay key is declared +`password: true` and is supplied by environment only. + +## Relationship to the other push stacks + +- [[systems/push-notifications]] documents the **Cadu/Vory** bridge (`hermes-push`, `vory-push`, + `~/.hermes/push/hermes_push.py`) — a **separate** stack, untouched by this plugin and still `degraded`. +- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the + verification table above). + +## Open + +- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk + item. +- The plugin carries a very large surface under one name; if it misbehaves the manifest `description` is the + only capability index. +- **Operational note:** the plugin's `pre_tool_call` guard rejects any tool call whose arguments contain the + literal text of a launchd one-off-restart command — writing *about* that command into a file is blocked + too. Describe it, don't quote it. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index a40b1a0..fbb6968 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-11-03 -verified_on: 2026-11-03 -review_after: 2026-11-10 +updated: 2026-11-04 +verified_on: 2026-11-04 +review_after: 2026-11-11 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + +- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` + + `deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` / + 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and + the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`** + after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26 + enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1) + joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`). + **Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, + 13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` + **03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and + again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` + untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`). +- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows; @@ -279,6 +291,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open +- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):** + `dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`** + (not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time + (`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) → + [[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are + both unrecovered.** - **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):** - **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed `status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent @@ -883,3 +901,64 @@ unidentified `-15` signal source; and the **30 s worker-ack window vs this host' **Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m. + + +## Re-verified 2026-11-04 (window 2026-11-03, AWST) + +**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1 +`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1** +(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new +terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session +compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** +after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at +13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. + +### Count (live, 2026-11-04) + +| Measure | 2026-11-03 window | Change vs 11-02 window | +|---|---|---| +| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged | +| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 | +| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** | +| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** | +| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 | +| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged | +| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged | +| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` | + +### What changed in the window + +- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the + 11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered + when it is `failed`. The cap-produced green row is the anomaly, not the red one. +- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no + handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30. +- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**, + `hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest + states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference + survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on + [[systems/push-notifications]], which is untouched and still `degraded`. +- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost + `error` did not reappear in this window. + +### Files touched by this audit + +- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`. +- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`, + `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update), + `log.md`. +- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and + `raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again. +- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no + decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window). + +### Still open (2026-11-04 — carries its predecessor forward) + +The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery +token; Camofox containment under multi-tab load; the Conduit health store with no retention; the +unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified +`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the +**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket; +whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should +**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no +handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing. diff --git a/systems/skills-index.md b/systems/skills-index.md index 30302ff..e92dc78 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-02 -verified_on: 2026-11-02 +updated: 2026-11-03 +verified_on: 2026-11-03 confidence: high tags: [skills, hermes] sources: []