wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log

This commit is contained in:
Hermes Wiki Audit
2026-11-04 00:16:23 +08:00
parent ca014efd63
commit 785fbc40b2
8 changed files with 534 additions and 13 deletions
@@ -3,11 +3,11 @@ title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete
type: incident
status: active
created: 2026-11-03
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-17
confidence: high
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
---
@@ -72,9 +72,11 @@ gap is that Stage 1 cannot finish inside its budget.
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered |
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
stage never reached.
stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a
different fault, and this time the ledger did go red; see the 2026-11-04 update below.**
## The "green row, no artefact" anti-pattern
@@ -98,6 +100,32 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
`cron_incidents` is not the same as success — here it is the *symptom*.
## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs
**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995
(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` /
`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`,
**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**.
→ [[incidents/2026-11-03-job-radar-stream-active-timeout]]
**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.**
When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself
kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02:
nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to
Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01
onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
stream active timeout."*
**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run
dies — the iteration cap is the specific case that still escapes as `completed`.
**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined
silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and
`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
unrecovered**, two days running.
## Open questions
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
@@ -112,6 +140,7 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
## Related
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[systems/scheduled-tasks]]
@@ -0,0 +1,118 @@
---
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
type: incident
status: open
created: 2026-11-04
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-11
confidence: high
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
---
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
## Headline
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
closed:
| Field | Value |
|---|---|
| `status` | **`failed`** |
| `error` | **`RuntimeError: stream active timeout`** |
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
## Timeline (all `+08:00`)
| Time | Event |
|---|---|
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
| 07:01:17 | run session `local-7180cef7…` first message |
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
| 13:28:51.741724 | **failure notice delivered** |
| 13:28:52.305995 | execution closes `failed` |
| 13:28:55.141618 | incident `alerted` |
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
## The 65-minute silence is the new information
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
executed.
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
provider's socket state during the silent window.
## Why the ledger went red this time
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|---|---|---|---|
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
a delivered notice; the cap produces a green row and silence. That is the distinction
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
| Slot | Job | Ran | Result |
|---|---|---|---|
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`).
## Context churn (observation, cause not established)
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
contributes to the stream stall is **not established** — this is recorded because both long failures share
the pattern, not because the link is proven.
## Open questions
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
silently long before the harness noticed?
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
re-slotted, or Stage 2 made to wait on a handoff?
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
day's discovery to the next slot?
## Related
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
- [[runbooks/job-radar]]
- [[systems/scheduled-tasks]]
+6 -2
View File
File diff suppressed because one or more lines are too long
+53
View File
@@ -1907,3 +1907,56 @@ recorded wrong.**
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
## [2026-11-04] audit — window 2026-11-03
**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` /
1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since
10-31**.
**New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1
(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on
**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35,
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at
**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident
`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still
open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse
carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream
active timeout"*).
**Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the
recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery
tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended
`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean
state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`.
Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.**
**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16,
14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
**New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author
*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`;
`config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the
replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory
remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched,
still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
blocked and had to be rephrased.
**Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:**
`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows,
7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39).
⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 =
**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts
uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this
audit committed them.
**Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop`
`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the
2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02).
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
+134
View File
@@ -0,0 +1,134 @@
# Wiki freshness audit — 2026-11-03
Threshold: >21 days since `updated`
## STALE (8)
- [ 23d] agents/qwenpaw.md (updated 2026-10-11)
- [ 23d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
- [ 23d] systems/ignis.md (updated 2026-10-11)
- [ 23d] systems/paseo.md (updated 2026-10-11)
- [ 22d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
- [ 22d] queries/retained-research-answers.md (updated 2026-10-12)
- [ 22d] systems/home-assistant.md (updated 2026-10-12)
- [ 22d] systems/nanobot.md (updated 2026-10-12)
## FRESH (110)
- [ 14d] CONTRIBUTING.md
- [ 8d] agents/hermes-chaos.md
- [ 8d] agents/hermes-production.md
- [ 14d] agents/index.md
- [ 5d] agents/nanobot.md
- [ 14d] agents/open-claw.md
- [ 14d] agents/zeroclaw.md
- [ 5d] comparisons/camoufox-vs-chrome.md
- [ 14d] comparisons/hermes-vs-nanobot.md
- [ 14d] comparisons/memory-backends.md
- [ 14d] comparisons/model-routers.md
- [ 5d] comparisons/steel-vs-camoufox.md
- [ 14d] concepts/agent-memory.md
- [ 5d] concepts/browser-automation.md
- [ 14d] concepts/model-routing.md
- [ 14d] concepts/persistent-shells.md
- [ 14d] concepts/tool-calling.md
- [ 3d] current-state.md
- [ 5d] decisions/index.md
- [ 14d] decisions/keep-openrouter-as-fallback.md
- [ 14d] decisions/persistent-shell-enabled.md
- [ 5d] decisions/rollback-steel-to-camofox.md
- [ 14d] experiments/index.md
- [ 10d] incidents/2026-07-29-mnemosyne-relay.md
- [ 14d] incidents/2026-08-01-mnemosyne-capture-failure.md
- [ 10d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
- [ 11d] incidents/2026-08-15-skills-index-drift.md
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 14d] incidents/2026-08-29-gmail-read-state.md
- [ 13d] incidents/2026-09-02-sftpgo-outage.md
- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 8d] incidents/2026-10-03-ct450-hermes-update.md
- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 10d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 2d] incidents/2026-10-09-memory-recall-relevance.md
- [ 18d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
- [ 7d] incidents/2026-10-16-omniroute-503-resource-pressure.md
- [ 15d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
- [ 7d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
- [ 12d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- [ 9d] incidents/2026-10-22-conduit-memory-start-timeout.md
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
- [ 8d] incidents/2026-10-24-conduit-call-request-refused.md
- [ 8d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
- [ 6d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
- [ 4d] incidents/2026-10-27-job-radar-budget-exhausted.md
- [ 4d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
- [ 4d] incidents/2026-10-29-conduit-voice-silence.md
- [ 3d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
- [ 4d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
- [ 3d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
- [ 2d] incidents/2026-10-31-asana-access-failures.md
- [ 2d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
- [ 2d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
- [ 1d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
- [ 0d] index.md
- [ 12d] infrastructure/backups.md
- [ 14d] infrastructure/credentials-map.md
- [ 2d] infrastructure/docker-services.md
- [ 14d] infrastructure/domains-and-tunnels.md
- [ 3d] infrastructure/hosts.md
- [ 20d] infrastructure/network-map.md
- [ 2d] infrastructure/proxmox.md
- [ 2d] infrastructure/tailscale.md
- [ 9d] reports/communities-wa-social-media-audit-2026.md
- [ 2d] reports/uber-eats/2026-07.md
- [ 2d] reports/uber-eats/2026-08.md
- [ 2d] reports/uber-eats/2026-09.md
- [ 2d] reports/uber-eats/2026-10.md
- [ 2d] reports/uber-eats/index.md
- [ 14d] runbooks/backup-wiki.md
- [ 14d] runbooks/diagnose-docker.md
- [ 14d] runbooks/gateway-resource-resilience.md
- [ 4d] runbooks/job-radar.md
- [ 14d] runbooks/multiplexer-setup.md
- [ 8d] runbooks/provider-health.md
- [ 5d] runbooks/recover-docker-service.md
- [ 14d] runbooks/restart-browser.md
- [ 14d] runbooks/restart-hermes.md
- [ 14d] runbooks/restore-openrouter.md
- [ 14d] runbooks/test-openrouter.md
- [ 14d] runbooks/troubleshoot-tool-loop.md
- [ 8d] runbooks/update-hermes-safely.md
- [ 0d] systems/browser-backend.md
- [ 14d] systems/cloudflare-tunnel.md
- [ 2d] systems/conduit.md
- [ 8d] systems/current-profile.md
- [ 14d] systems/guanaco.md
- [ 14d] systems/headless-server.md
- [ 4d] systems/hermes-agent.md
- [ 14d] systems/hermes-ui.md
- [ 4d] systems/homepage-dashboard.md
- [ 8d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 2d] systems/mnemosyne.md
- [ 8d] systems/model-providers.md
- [ 14d] systems/nextcloud.md
- [ 14d] systems/obsidian.md
- [ 2d] systems/omniroute.md
- [ 14d] systems/opencode-go.md
- [ 14d] systems/openrouter.md
- [ 15d] systems/push-notifications.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
- [ 14d] systems/terminal-backend.md
- [ 14d] systems/tool-search.md
## NO updated DATE (3)
- SCHEMA.md
- experiments/active/example.md
- log.md
TOTAL: 8 stale, 110 fresh, 3 no-date of 121 pages
+104
View File
@@ -0,0 +1,104 @@
---
title: hermes-hq-push (Hermes HQ iPhone app plugin)
type: system
status: active
created: 2026-11-04
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-12-04
confidence: medium
tags: [push, notifications, plugins, ios, desktop, device-access]
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
---
# hermes-hq-push (Hermes HQ iPhone app plugin)
## What it is
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*)
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half
of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal
page, and Grok Live voice token minting.
Its own manifest states it **replaces `dispatch-push`**, and that *"its registered phones carry over."*
## Verified state (2026-11-04)
| Check | Result |
|---|---|
| `hermes plugins list` | `hermes-hq-push` — **enabled**, `0.9.27`, state `user` |
| `config.yaml` → `plugins.enabled` | line **7556**: `- hermes-hq-push` |
| `~/.hermes/plugins/dispatch-push/` | **absent** — and no `dispatch-push` reference survives in `config.yaml` |
| live tree | `~/.hermes/plugins/hermes-hq-push/` (module files + `dashboard/` subdir + its own test suite) |
| README | none — the manifest `description` is the only prose |
The replacement is therefore **clean**: nothing references the retired plugin, and the new one is explicitly
enabled. Enablement and version are config-verified; individual hook firing was not separately captured.
## Surfaces (grouped from the plugin manifest)
**Push / approvals**
- Finished turns — who replied and what they said.
- Approvals with the command shown, **Approve / Deny from the lock screen**.
- A bot waiting for a password, login, code or answer — the site or the question, **never the value**.
- **Live Lock Screen cards:** every step the bot takes (e.g. *Running npm test…*, the two before it, how
many) is pushed to the card, *sealed so only the phone can read it*; the card says **Done** when the turn
ends.
- Tells the app which scheduled jobs a bot is still working on.
**Device access** (app *Settings › Device Access*) — contacts lookup, location, reminders and events, alarms
and timers, photos, Health, motion, music, and leaving an email or text for Anthony to send. This is what
backs the Conduit-style phone sharing on the Hermes HQ app.
**Desktop app** — adds the **Hermes HQ theme**; opens a shell on the host for the app's Terminal page (the
login shell, as an SSH login would; `plugin-data`'s `terminal.off` disables it); lets a Nous sign-in delete,
archive and rename the chats it created (Hermes refuses them otherwise) and lets the desktop app open chats
the app made (Hermes answers `not_found` otherwise); keeps a bot's **Sessions list fast** on a host that has
run many scheduled jobs; lets the app send into a chat the desktop app has open; page previews; and keeps
the app's performance log on this computer at **`~/.hermes/hq-logs`**.
**Voice** — mints the app's **Grok Live** voice session a short-lived xAI token (subscription sign-in or
`XAI_API_KEY`; the bearer itself never reaches the phone).
**Restart guard** — stops a bot's one-off macOS launchd restart job from looping (those jobs are KeepAlive,
so a one-time restart re-runs forever and locks the app out), and tells a restart script which in-app chats
are mid-reply so a restart can wait.
**Cross-posting** — on request, posts the bot's answer back to the Telegram or Discord chat a conversation
came from.
## Interface
- **Tools:** `device_contacts_search`, `device_request_result`.
- **Hooks:** `pre_approval_request`, `post_approval_response`, `pre_tool_call`, `on_human_input_request`,
`on_human_input_resolved`, `pre_llm_call`, `on_session_end`.
## Configuration
All environment variables are optional. `DISPATCH_*` spellings of the APNs/relay names still work.
| Variable | Purpose |
|---|---|
| `HERMES_HQ_APNS_KEY_PATH`, `HERMES_HQ_APNS_KEY_ID`, `HERMES_HQ_APNS_TEAM_ID` | **Direct APNs mode** — path to the app's `.p8` auth key, its Key ID, and the Apple Developer Team ID that owns the app. *Only the app's developer has these.* |
| `HERMES_HQ_PUSH_RELAY_URL` | Another push relay's HTTPS URL, or `off` to send nothing. **Unset = alerts go through the public Hermes HQ relay, sealed so it cannot read them.** |
| `HERMES_HQ_PUSH_RELAY_KEY` | Only for a private relay that asks for a key. |
No key material is recorded on this page and none appears in the manifest — the relay key is declared
`password: true` and is supplied by environment only.
## Relationship to the other push stacks
- [[systems/push-notifications]] documents the **Cadu/Vory** bridge (`hermes-push`, `vory-push`,
`~/.hermes/push/hermes_push.py`) — a **separate** stack, untouched by this plugin and still `degraded`.
- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the
verification table above).
## Open
- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk
item.
- The plugin carries a very large surface under one name; if it misbehaves the manifest `description` is the
only capability index.
- **Operational note:** the plugin's `pre_tool_call` guard rejects any tool call whose arguments contain the
literal text of a launchd one-off-restart command — writing *about* that command into a file is blocked
too. Describe it, don't quote it.
+83 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-11
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`**
after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26
enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1)
joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`).
**Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**,
13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271`
**03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and
again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt`
untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`).
- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
@@ -279,6 +291,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):**
`dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`**
(not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
both unrecovered.**
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
@@ -883,3 +901,64 @@ unidentified `-15` signal source; and the **30 s worker-ack window vs this host'
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
## Re-verified 2026-11-04 (window 2026-11-03, AWST)
**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1
`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1**
(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new
terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session
compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s**
after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at
13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]].
### Count (live, 2026-11-04)
| Measure | 2026-11-03 window | Change vs 11-02 window |
|---|---|---|
| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged |
| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 |
| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** |
| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** |
| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 |
| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged |
| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged |
| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` |
### What changed in the window
- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the
11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered
when it is `failed`. The cap-produced green row is the anomaly, not the red one.
- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no
handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**,
`hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest
states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference
survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on
[[systems/push-notifications]], which is untouched and still `degraded`.
- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost
`error` did not reappear in this window.
### Files touched by this audit
- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`,
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update),
`log.md`.
- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and
`raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again.
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window).
### Still open (2026-11-04 — carries its predecessor forward)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified
`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the
**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket;
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-11-02
verified_on: 2026-11-02
updated: 2026-11-03
verified_on: 2026-11-03
confidence: high
tags: [skills, hermes]
sources: []