wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log
This commit is contained in:
@@ -3,11 +3,11 @@ title: Job Radar Stage 1 Hit the Iteration Cap Again (2026-11-02) — Incomplete
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-03
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-17
|
||||
confidence: high
|
||||
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox]
|
||||
tags: [cron, job-radar, iteration-cap, silent-failure, notion, camofox, stream-timeout]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-state-db, live-jobs-json, session-history]
|
||||
---
|
||||
|
||||
@@ -72,9 +72,11 @@ gap is that Stage 1 cannot finish inside its budget.
|
||||
| 2026-10-27 | [[incidents/2026-10-27-job-radar-budget-exhausted]] | Tool-call budget exhausted before the Notion write stage |
|
||||
| 2026-10-28 | [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] | Stage 1 hit the agent iteration cap before the handoff |
|
||||
| **2026-11-02** | **this page** | Same cap; **run recorded `completed` and its report suppressed** |
|
||||
| 2026-11-03 | [[incidents/2026-11-03-job-radar-stream-active-timeout]] | **A different ceiling** — `RuntimeError: stream active timeout` after 6 h 28 m; `failed`, notice delivered |
|
||||
|
||||
Three hits in seven days, each ending in the same place: discovery complete, handoff absent, Notion write
|
||||
stage never reached.
|
||||
stage never reached. **A fourth run — 2026-11-03 — ended the same way (no handoff, both Stage 2 slots declined) but on a
|
||||
different fault, and this time the ledger did go red; see the 2026-11-04 update below.**
|
||||
|
||||
## The "green row, no artefact" anti-pattern
|
||||
|
||||
@@ -98,6 +100,32 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
|
||||
`cron_incidents` stayed quiet for a reason, before treating the slot as served. Silence from
|
||||
`cron_incidents` is not the same as success — here it is the *symptom*.
|
||||
|
||||
## 2026-11-04 update — the missing incident row was a property of the cap, not of long runs
|
||||
|
||||
**Job Radar Stage 1 lost a second consecutive day on 2026-11-03 — and this time the harness killed the
|
||||
turn, so the ledger went red.** The 07:00 `dca8482e4f76` ran **07:00:33.819190 → 13:28:52.305995
|
||||
(6 h 28 m 18 s)** and closed **`status: failed` / `error: RuntimeError: stream active timeout` /
|
||||
`delivery_outcome: delivered`**, raising incident **`dca848_dfd4bea84f2a`** (`failure_type: timeout`,
|
||||
**`alerted` 13:28:55.141618, still open**) — fleet unclosed rows **0 → 1**.
|
||||
→ [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
|
||||
**This answers open question 3 below (partly): suppression tracked the terminal state, not the run length.**
|
||||
When the run ends `completed` — as the cap produces — its report is suppressed; when the harness itself
|
||||
kills the turn, the terminal state is `failed` and both the incident row and the notice appear. 11-02:
|
||||
nothing delivered. 11-03: the failure notice went out at **13:28:51.741724** and Pulse surfaced it to
|
||||
Anthony in the desktop session `20261031_190813_95fbde` (report `p34c8e899-1`, covering Tue 03 Nov 13:01
|
||||
onward): *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a
|
||||
stream active timeout."*
|
||||
|
||||
**It also narrows open question 2 below:** the ledger *can* record a non-`completed` state when a long run
|
||||
dies — the iteration cap is the specific case that still escapes as `completed`.
|
||||
|
||||
**And it partially answers open question 4: the candidate set was lost again.** Both Stage 2 slots declined
|
||||
silently (`5fc292ff0f3b` 11:30:36 → 11:32:16 and 14:30:32 → 14:33:11) and
|
||||
`~/.hermes/radar-ref/handoff/` still holds **only** `2026-10-27.json`, `2026-10-29.json` and
|
||||
`2026-10-30.json` — so **2026-11-02's 164 deduped candidates and 2026-11-03's discovery are both
|
||||
unrecovered**, two days running.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Is the iteration budget the right control, or is the task too large for one agent turn?** Three caps in
|
||||
@@ -112,6 +140,7 @@ The 2026-11-02 cap raised **nothing at all** (`cron_incidents` holds no row for
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-03-job-radar-stream-active-timeout]]
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, job-radar, stream-timeout, silent-failure, notion, context-compaction]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-deliveries-db, live-jobs-json, live-state-db, live-gateway-log, live-radar-handoff-dir, session-history]
|
||||
---
|
||||
|
||||
# Job Radar Stage 1 Died on a Provider Stream Timeout (2026-11-03) — Second Consecutive Day Without a Handoff
|
||||
|
||||
## Headline
|
||||
|
||||
**The Stage-1 slot lost a second consecutive day — and this time the harness killed the turn, so the
|
||||
ledger went red.** On **2026-11-03** the 07:00 `dca8482e4f76` (*Daily Marketing Job Radar — Stage 1
|
||||
(Discovery)*) ran **6 h 28 m 18 s** (`started 07:00:33.819190` → `finished 13:28:52.305995 +08:00`) and
|
||||
closed:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| `status` | **`failed`** |
|
||||
| `error` | **`RuntimeError: stream active timeout`** |
|
||||
| `delivery_outcome` | **`delivered`** (the failure notice, `for_failure: 1`) |
|
||||
| `cron_incidents` row | **`dca848_dfd4bea84f2a`**, `failure_type: timeout`, **`alerted`**, still open |
|
||||
|
||||
No handoff artefact was written, so **nothing reached Notion, no cover letters were generated, and both
|
||||
Stage 2 slots declined.** This is the **fourth Stage-1 failure in eight days** and a **new terminal
|
||||
signature** — not the iteration cap that ended 10-27, 10-28 and 11-02.
|
||||
|
||||
## Timeline (all `+08:00`)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 07:00:33.819190 | execution `dca8482e4f76` opens (07:00 weekday slot) |
|
||||
| 07:01:17 | run session `local-7180cef7…` first message |
|
||||
| 08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21 | **five context compactions** in one run |
|
||||
| 11:30:36.617545 → 11:32:16.812897 | Stage 2 `5fc292ff0f3b` first slot: no handoff → `[SILENT]`, `suppressed` |
|
||||
| 12:23:28 | **last tool result of the run** (a `terminal` call) |
|
||||
| 13:28:49 | harness records *"This turn did not complete. Some actions may already have run; verify their effects before resending."* |
|
||||
| 13:28:50,627 | gateway.log: `ERROR gateway.session_authority: Admitted turn 9bf9a296… failed` (traceback through `session_authority.py:1159 _execute_claimed` → `session_finite.py:80 execute_finite_admission` → `session_ingress.py:235 execute_admission`) |
|
||||
| 13:28:51.463929 | incident `dca848_dfd4bea84f2a` opens |
|
||||
| 13:28:51.741724 | **failure notice delivered** |
|
||||
| 13:28:52.305995 | execution closes `failed` |
|
||||
| 13:28:55.141618 | incident `alerted` |
|
||||
| 14:30:32.261931 → 14:33:11.470879 | Stage 2 safety-net slot: same silent decline |
|
||||
| in-session | Pulse report `p34c8e899-1` (covering Tue 03 Nov 13:01 onward) led with it: *"The Daily Marketing Job Radar — Discovery job is now failing, with its last run dying on a stream active timeout."* |
|
||||
|
||||
## The 65-minute silence is the new information
|
||||
|
||||
The run's recorded activity stops at **12:23:28** and the harness does not declare the turn dead until
|
||||
**13:28:49** — **65 m 21 s with no tool call, no model turn and no error written to the ledger**.
|
||||
|
||||
Contrast with the two cap failures: on **10-28** and **11-02** the runtime *injected* the text *"You've
|
||||
reached the maximum number of tool-calling iterations allowed…"* into the session before it ended. This
|
||||
session contains **no such message** — it stopped mid-verification, on a provider-stream liveness timeout,
|
||||
with the run's own closing plan (*"Two final verification passes, then I'll assemble the handoff"*) never
|
||||
executed.
|
||||
|
||||
**Therefore:** the agent iteration cap is not the only ceiling that ends a long Stage-1 run, and the
|
||||
"2026-11-02 raised no incident at all" behaviour was a property of *that* terminal state (`completed`,
|
||||
`error: NULL`), **not** of long runs in general. Nothing in the ledger or the gateway log records the
|
||||
provider's socket state during the silent window.
|
||||
|
||||
## Why the ledger went red this time
|
||||
|
||||
| Date | `executions.status` | `cron_incidents` row | Notice to Anthony |
|
||||
|---|---|---|---|
|
||||
| 2026-10-27 | non-clean (`failed`) | `dca848_e93a538ade6b` — `failure_type: auth` (**wrong** for a budget fault) | yes — the run's INCOMPLETE report |
|
||||
| 2026-10-28 | non-clean (`failed`) | `dca848_0a2f03b5952b` — `failure_type: timeout` | yes |
|
||||
| 2026-11-02 | **`completed`, `error: NULL`** | **none raised** | **no — `suppressed`** |
|
||||
| **2026-11-03** | **`failed`** | **`dca848_dfd4bea84f2a` — `timeout`, open** | **yes — 13:28:51.741724** |
|
||||
|
||||
Fleet-wide **unclosed incident rows 0 → 1**. A run the harness itself kills produces an honest red row and
|
||||
a delivered notice; the cap produces a green row and silence. That is the distinction
|
||||
[[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] could only predict, now observed.
|
||||
|
||||
## Downstream: both Stage 2 slots behaved correctly, and two days of candidates are lost
|
||||
|
||||
| Slot | Job | Ran | Result |
|
||||
|---|---|---|---|
|
||||
| 11:30 | `5fc292ff0f3b` (*Write and Letters*) | 11:30:36 → 11:32:16 | `[SILENT]` / `suppressed` |
|
||||
| 14:30 (safety net) | `5fc292ff0f3b` | 14:30:32 → 14:33:11 | `[SILENT]` / `suppressed` |
|
||||
|
||||
`~/.hermes/radar-ref/handoff/` holds **only** `2026-10-27.json`, `2026-10-29.json` and `2026-10-30.json` —
|
||||
nothing for 10-31, **11-02** or **11-03**. So **2026-11-02's 164 deduped candidates and 2026-11-03's
|
||||
discovery are both unrecovered**, and Notion received nothing on either day. The last end-to-end success
|
||||
remains **2026-10-29** (handoff written inside budget; Stage 2 landed **20 pages + 2 cover letters**,
|
||||
`missing_cover_letter_count 0`).
|
||||
|
||||
## Context churn (observation, cause not established)
|
||||
|
||||
**363 messages** (6 `user` / 133 `assistant` / 224 `tool`), **288 compaction-flagged messages**, and **five
|
||||
compactions** inside one 6 h run. The 11-02 run compacted repeatedly as well. Whether compaction churn
|
||||
contributes to the stream stall is **not established** — this is recorded because both long failures share
|
||||
the pattern, not because the link is proven.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Is the **65 m 21 s** silence the stream-active watchdog's own threshold, or did the provider socket die
|
||||
silently long before the harness noticed?
|
||||
2. Does repeated context compaction contribute to the stall? Two long runs, both compacting heavily.
|
||||
3. The 07:00 run has now overrun past the 11:30 Stage-2 slot two days running — should the pairing be
|
||||
re-slotted, or Stage 2 made to wait on a handoff?
|
||||
4. Where does `radar-ref/handoff/` get its retention? The newest file is 10-30.
|
||||
5. Should a stream timeout on a *file-deliverable* job trigger an automatic retry instead of leaving the
|
||||
day's discovery to the next slot?
|
||||
|
||||
## Related
|
||||
|
||||
- [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]
|
||||
- [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]
|
||||
- [[incidents/2026-10-27-job-radar-budget-exhausted]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
@@ -1907,3 +1907,56 @@ recorded wrong.**
|
||||
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
|
||||
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
|
||||
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
|
||||
|
||||
|
||||
## [2026-11-04] audit — window 2026-11-03
|
||||
|
||||
**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` /
|
||||
1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since
|
||||
10-31**.
|
||||
|
||||
**New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1
|
||||
(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on
|
||||
**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35,
|
||||
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at
|
||||
**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident
|
||||
`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still
|
||||
open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse
|
||||
carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream
|
||||
active timeout"*).
|
||||
|
||||
**Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the
|
||||
recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery
|
||||
tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended
|
||||
`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean
|
||||
state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`.
|
||||
Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.**
|
||||
|
||||
**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16,
|
||||
14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
|
||||
|
||||
**New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author
|
||||
*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`;
|
||||
`config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the
|
||||
replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory
|
||||
remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched,
|
||||
still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call
|
||||
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
|
||||
blocked and had to be rephrased.
|
||||
|
||||
**Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:**
|
||||
`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows,
|
||||
7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39).
|
||||
|
||||
⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 =
|
||||
**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts
|
||||
uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this
|
||||
audit committed them.
|
||||
|
||||
**Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop`
|
||||
`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the
|
||||
2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02).
|
||||
|
||||
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||||
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
|
||||
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Wiki freshness audit — 2026-11-03
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (8)
|
||||
|
||||
- [ 23d] agents/qwenpaw.md (updated 2026-10-11)
|
||||
- [ 23d] incidents/2026-09-28-hermes-managed-runtime-deps.md (updated 2026-10-11)
|
||||
- [ 23d] systems/ignis.md (updated 2026-10-11)
|
||||
- [ 23d] systems/paseo.md (updated 2026-10-11)
|
||||
- [ 22d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-10-12)
|
||||
- [ 22d] queries/retained-research-answers.md (updated 2026-10-12)
|
||||
- [ 22d] systems/home-assistant.md (updated 2026-10-12)
|
||||
- [ 22d] systems/nanobot.md (updated 2026-10-12)
|
||||
|
||||
## FRESH (110)
|
||||
|
||||
- [ 14d] CONTRIBUTING.md
|
||||
- [ 8d] agents/hermes-chaos.md
|
||||
- [ 8d] agents/hermes-production.md
|
||||
- [ 14d] agents/index.md
|
||||
- [ 5d] agents/nanobot.md
|
||||
- [ 14d] agents/open-claw.md
|
||||
- [ 14d] agents/zeroclaw.md
|
||||
- [ 5d] comparisons/camoufox-vs-chrome.md
|
||||
- [ 14d] comparisons/hermes-vs-nanobot.md
|
||||
- [ 14d] comparisons/memory-backends.md
|
||||
- [ 14d] comparisons/model-routers.md
|
||||
- [ 5d] comparisons/steel-vs-camoufox.md
|
||||
- [ 14d] concepts/agent-memory.md
|
||||
- [ 5d] concepts/browser-automation.md
|
||||
- [ 14d] concepts/model-routing.md
|
||||
- [ 14d] concepts/persistent-shells.md
|
||||
- [ 14d] concepts/tool-calling.md
|
||||
- [ 3d] current-state.md
|
||||
- [ 5d] decisions/index.md
|
||||
- [ 14d] decisions/keep-openrouter-as-fallback.md
|
||||
- [ 14d] decisions/persistent-shell-enabled.md
|
||||
- [ 5d] decisions/rollback-steel-to-camofox.md
|
||||
- [ 14d] experiments/index.md
|
||||
- [ 10d] incidents/2026-07-29-mnemosyne-relay.md
|
||||
- [ 14d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||
- [ 10d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- [ 12d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||
- [ 11d] incidents/2026-08-15-skills-index-drift.md
|
||||
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||
- [ 14d] incidents/2026-08-29-gmail-read-state.md
|
||||
- [ 13d] incidents/2026-09-02-sftpgo-outage.md
|
||||
- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 8d] incidents/2026-10-03-ct450-hermes-update.md
|
||||
- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||
- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||
- [ 10d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||
- [ 2d] incidents/2026-10-09-memory-recall-relevance.md
|
||||
- [ 18d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||
- [ 7d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||
- [ 15d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||
- [ 7d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||
- [ 12d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||
- [ 9d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||
- [ 8d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||
- [ 8d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||
- [ 8d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||
- [ 6d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||
- [ 4d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||
- [ 4d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||
- [ 4d] incidents/2026-10-29-conduit-voice-silence.md
|
||||
- [ 3d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||
- [ 4d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||
- [ 3d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||
- [ 2d] incidents/2026-10-31-asana-access-failures.md
|
||||
- [ 2d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||
- [ 2d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||
- [ 1d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||
- [ 0d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||
- [ 0d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||
- [ 0d] index.md
|
||||
- [ 12d] infrastructure/backups.md
|
||||
- [ 14d] infrastructure/credentials-map.md
|
||||
- [ 2d] infrastructure/docker-services.md
|
||||
- [ 14d] infrastructure/domains-and-tunnels.md
|
||||
- [ 3d] infrastructure/hosts.md
|
||||
- [ 20d] infrastructure/network-map.md
|
||||
- [ 2d] infrastructure/proxmox.md
|
||||
- [ 2d] infrastructure/tailscale.md
|
||||
- [ 9d] reports/communities-wa-social-media-audit-2026.md
|
||||
- [ 2d] reports/uber-eats/2026-07.md
|
||||
- [ 2d] reports/uber-eats/2026-08.md
|
||||
- [ 2d] reports/uber-eats/2026-09.md
|
||||
- [ 2d] reports/uber-eats/2026-10.md
|
||||
- [ 2d] reports/uber-eats/index.md
|
||||
- [ 14d] runbooks/backup-wiki.md
|
||||
- [ 14d] runbooks/diagnose-docker.md
|
||||
- [ 14d] runbooks/gateway-resource-resilience.md
|
||||
- [ 4d] runbooks/job-radar.md
|
||||
- [ 14d] runbooks/multiplexer-setup.md
|
||||
- [ 8d] runbooks/provider-health.md
|
||||
- [ 5d] runbooks/recover-docker-service.md
|
||||
- [ 14d] runbooks/restart-browser.md
|
||||
- [ 14d] runbooks/restart-hermes.md
|
||||
- [ 14d] runbooks/restore-openrouter.md
|
||||
- [ 14d] runbooks/test-openrouter.md
|
||||
- [ 14d] runbooks/troubleshoot-tool-loop.md
|
||||
- [ 8d] runbooks/update-hermes-safely.md
|
||||
- [ 0d] systems/browser-backend.md
|
||||
- [ 14d] systems/cloudflare-tunnel.md
|
||||
- [ 2d] systems/conduit.md
|
||||
- [ 8d] systems/current-profile.md
|
||||
- [ 14d] systems/guanaco.md
|
||||
- [ 14d] systems/headless-server.md
|
||||
- [ 4d] systems/hermes-agent.md
|
||||
- [ 14d] systems/hermes-ui.md
|
||||
- [ 4d] systems/homepage-dashboard.md
|
||||
- [ 8d] systems/memory-backend.md
|
||||
- [ 1d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 8d] systems/model-providers.md
|
||||
- [ 14d] systems/nextcloud.md
|
||||
- [ 14d] systems/obsidian.md
|
||||
- [ 2d] systems/omniroute.md
|
||||
- [ 14d] systems/opencode-go.md
|
||||
- [ 14d] systems/openrouter.md
|
||||
- [ 15d] systems/push-notifications.md
|
||||
- [ 0d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
- [ 14d] systems/terminal-backend.md
|
||||
- [ 14d] systems/tool-search.md
|
||||
|
||||
## NO updated DATE (3)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- log.md
|
||||
|
||||
TOTAL: 8 stale, 110 fresh, 3 no-date of 121 pages
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: hermes-hq-push (Hermes HQ iPhone app plugin)
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-12-04
|
||||
confidence: medium
|
||||
tags: [push, notifications, plugins, ios, desktop, device-access]
|
||||
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
|
||||
---
|
||||
|
||||
# hermes-hq-push (Hermes HQ iPhone app plugin)
|
||||
|
||||
## What it is
|
||||
|
||||
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*)
|
||||
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half
|
||||
of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal
|
||||
page, and Grok Live voice token minting.
|
||||
|
||||
Its own manifest states it **replaces `dispatch-push`**, and that *"its registered phones carry over."*
|
||||
|
||||
## Verified state (2026-11-04)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `hermes plugins list` | `hermes-hq-push` — **enabled**, `0.9.27`, state `user` |
|
||||
| `config.yaml` → `plugins.enabled` | line **7556**: `- hermes-hq-push` |
|
||||
| `~/.hermes/plugins/dispatch-push/` | **absent** — and no `dispatch-push` reference survives in `config.yaml` |
|
||||
| live tree | `~/.hermes/plugins/hermes-hq-push/` (module files + `dashboard/` subdir + its own test suite) |
|
||||
| README | none — the manifest `description` is the only prose |
|
||||
|
||||
The replacement is therefore **clean**: nothing references the retired plugin, and the new one is explicitly
|
||||
enabled. Enablement and version are config-verified; individual hook firing was not separately captured.
|
||||
|
||||
## Surfaces (grouped from the plugin manifest)
|
||||
|
||||
**Push / approvals**
|
||||
- Finished turns — who replied and what they said.
|
||||
- Approvals with the command shown, **Approve / Deny from the lock screen**.
|
||||
- A bot waiting for a password, login, code or answer — the site or the question, **never the value**.
|
||||
- **Live Lock Screen cards:** every step the bot takes (e.g. *Running npm test…*, the two before it, how
|
||||
many) is pushed to the card, *sealed so only the phone can read it*; the card says **Done** when the turn
|
||||
ends.
|
||||
- Tells the app which scheduled jobs a bot is still working on.
|
||||
|
||||
**Device access** (app *Settings › Device Access*) — contacts lookup, location, reminders and events, alarms
|
||||
and timers, photos, Health, motion, music, and leaving an email or text for Anthony to send. This is what
|
||||
backs the Conduit-style phone sharing on the Hermes HQ app.
|
||||
|
||||
**Desktop app** — adds the **Hermes HQ theme**; opens a shell on the host for the app's Terminal page (the
|
||||
login shell, as an SSH login would; `plugin-data`'s `terminal.off` disables it); lets a Nous sign-in delete,
|
||||
archive and rename the chats it created (Hermes refuses them otherwise) and lets the desktop app open chats
|
||||
the app made (Hermes answers `not_found` otherwise); keeps a bot's **Sessions list fast** on a host that has
|
||||
run many scheduled jobs; lets the app send into a chat the desktop app has open; page previews; and keeps
|
||||
the app's performance log on this computer at **`~/.hermes/hq-logs`**.
|
||||
|
||||
**Voice** — mints the app's **Grok Live** voice session a short-lived xAI token (subscription sign-in or
|
||||
`XAI_API_KEY`; the bearer itself never reaches the phone).
|
||||
|
||||
**Restart guard** — stops a bot's one-off macOS launchd restart job from looping (those jobs are KeepAlive,
|
||||
so a one-time restart re-runs forever and locks the app out), and tells a restart script which in-app chats
|
||||
are mid-reply so a restart can wait.
|
||||
|
||||
**Cross-posting** — on request, posts the bot's answer back to the Telegram or Discord chat a conversation
|
||||
came from.
|
||||
|
||||
## Interface
|
||||
|
||||
- **Tools:** `device_contacts_search`, `device_request_result`.
|
||||
- **Hooks:** `pre_approval_request`, `post_approval_response`, `pre_tool_call`, `on_human_input_request`,
|
||||
`on_human_input_resolved`, `pre_llm_call`, `on_session_end`.
|
||||
|
||||
## Configuration
|
||||
|
||||
All environment variables are optional. `DISPATCH_*` spellings of the APNs/relay names still work.
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `HERMES_HQ_APNS_KEY_PATH`, `HERMES_HQ_APNS_KEY_ID`, `HERMES_HQ_APNS_TEAM_ID` | **Direct APNs mode** — path to the app's `.p8` auth key, its Key ID, and the Apple Developer Team ID that owns the app. *Only the app's developer has these.* |
|
||||
| `HERMES_HQ_PUSH_RELAY_URL` | Another push relay's HTTPS URL, or `off` to send nothing. **Unset = alerts go through the public Hermes HQ relay, sealed so it cannot read them.** |
|
||||
| `HERMES_HQ_PUSH_RELAY_KEY` | Only for a private relay that asks for a key. |
|
||||
|
||||
No key material is recorded on this page and none appears in the manifest — the relay key is declared
|
||||
`password: true` and is supplied by environment only.
|
||||
|
||||
## Relationship to the other push stacks
|
||||
|
||||
- [[systems/push-notifications]] documents the **Cadu/Vory** bridge (`hermes-push`, `vory-push`,
|
||||
`~/.hermes/push/hermes_push.py`) — a **separate** stack, untouched by this plugin and still `degraded`.
|
||||
- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the
|
||||
verification table above).
|
||||
|
||||
## Open
|
||||
|
||||
- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk
|
||||
item.
|
||||
- The plugin carries a very large surface under one name; if it misbehaves the manifest `description` is the
|
||||
only capability index.
|
||||
- **Operational note:** the plugin's `pre_tool_call` guard rejects any tool call whose arguments contain the
|
||||
literal text of a launchd one-off-restart command — writing *about* that command into a file is blocked
|
||||
too. Describe it, don't quote it.
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
|
||||
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
|
||||
the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`**
|
||||
after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26
|
||||
enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1)
|
||||
joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`).
|
||||
**Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**,
|
||||
13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271`
|
||||
**03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and
|
||||
again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt`
|
||||
untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`).
|
||||
- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
|
||||
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
|
||||
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
|
||||
@@ -279,6 +291,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||||
|
||||
### Open
|
||||
- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):**
|
||||
`dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`**
|
||||
(not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time
|
||||
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
|
||||
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
|
||||
both unrecovered.**
|
||||
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
|
||||
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
|
||||
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
|
||||
@@ -883,3 +901,64 @@ unidentified `-15` signal source; and the **30 s worker-ack window vs this host'
|
||||
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
|
||||
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
|
||||
|
||||
|
||||
|
||||
## Re-verified 2026-11-04 (window 2026-11-03, AWST)
|
||||
|
||||
**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1
|
||||
`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1**
|
||||
(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new
|
||||
terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session
|
||||
compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s**
|
||||
after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at
|
||||
13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]].
|
||||
|
||||
### Count (live, 2026-11-04)
|
||||
|
||||
| Measure | 2026-11-03 window | Change vs 11-02 window |
|
||||
|---|---|---|
|
||||
| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged |
|
||||
| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 |
|
||||
| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** |
|
||||
| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** |
|
||||
| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 |
|
||||
| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged |
|
||||
| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged |
|
||||
| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` |
|
||||
|
||||
### What changed in the window
|
||||
|
||||
- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the
|
||||
11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered
|
||||
when it is `failed`. The cap-produced green row is the anomaly, not the red one.
|
||||
- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no
|
||||
handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
|
||||
- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**,
|
||||
`hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest
|
||||
states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference
|
||||
survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on
|
||||
[[systems/push-notifications]], which is untouched and still `degraded`.
|
||||
- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost
|
||||
`error` did not reappear in this window.
|
||||
|
||||
### Files touched by this audit
|
||||
|
||||
- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`.
|
||||
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`,
|
||||
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update),
|
||||
`log.md`.
|
||||
- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and
|
||||
`raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again.
|
||||
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window).
|
||||
|
||||
### Still open (2026-11-04 — carries its predecessor forward)
|
||||
|
||||
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
|
||||
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
|
||||
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified
|
||||
`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the
|
||||
**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket;
|
||||
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
|
||||
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
|
||||
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user