wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log
This commit is contained in:
@@ -1907,3 +1907,56 @@ recorded wrong.**
|
||||
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
|
||||
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
|
||||
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
|
||||
|
||||
|
||||
## [2026-11-04] audit — window 2026-11-03
|
||||
|
||||
**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` /
|
||||
1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since
|
||||
10-31**.
|
||||
|
||||
**New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1
|
||||
(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on
|
||||
**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35,
|
||||
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at
|
||||
**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident
|
||||
`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still
|
||||
open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse
|
||||
carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream
|
||||
active timeout"*).
|
||||
|
||||
**Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the
|
||||
recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery
|
||||
tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended
|
||||
`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean
|
||||
state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`.
|
||||
Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.**
|
||||
|
||||
**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16,
|
||||
14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
|
||||
|
||||
**New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author
|
||||
*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`;
|
||||
`config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the
|
||||
replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory
|
||||
remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched,
|
||||
still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call
|
||||
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
|
||||
blocked and had to be rephrased.
|
||||
|
||||
**Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:**
|
||||
`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows,
|
||||
7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39).
|
||||
|
||||
⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 =
|
||||
**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts
|
||||
uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this
|
||||
audit committed them.
|
||||
|
||||
**Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop`
|
||||
`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the
|
||||
2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02).
|
||||
|
||||
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||||
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
|
||||
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
|
||||
|
||||
Reference in New Issue
Block a user