wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log

This commit is contained in:
Hermes Wiki Audit
2026-11-04 00:16:23 +08:00
parent ca014efd63
commit 785fbc40b2
8 changed files with 534 additions and 13 deletions
+83 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-03
verified_on: 2026-11-03
review_after: 2026-11-10
updated: 2026-11-04
verified_on: 2026-11-04
review_after: 2026-11-11
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`**
after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26
enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1)
joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`).
**Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**,
13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271`
**03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and
again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt`
untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`).
- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
@@ -279,6 +291,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):**
`dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`**
(not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
both unrecovered.**
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
@@ -883,3 +901,64 @@ unidentified `-15` signal source; and the **30 s worker-ack window vs this host'
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
## Re-verified 2026-11-04 (window 2026-11-03, AWST)
**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1
`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1**
(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new
terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session
compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s**
after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at
13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]].
### Count (live, 2026-11-04)
| Measure | 2026-11-03 window | Change vs 11-02 window |
|---|---|---|
| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged |
| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 |
| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** |
| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** |
| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 |
| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged |
| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged |
| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` |
### What changed in the window
- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the
11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered
when it is `failed`. The cap-produced green row is the anomaly, not the red one.
- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no
handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**,
`hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest
states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference
survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on
[[systems/push-notifications]], which is untouched and still `degraded`.
- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost
`error` did not reappear in this window.
### Files touched by this audit
- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`,
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update),
`log.md`.
- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and
`raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again.
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window).
### Still open (2026-11-04 — carries its predecessor forward)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified
`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the
**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket;
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.