audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129
This commit is contained in:
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-07
|
||||
verified_on: 2026-11-07
|
||||
review_after: 2026-11-08
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-11-09
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -1155,3 +1155,82 @@ completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
|
||||
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
|
||||
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed.
|
||||
|
||||
## Re-verified 2026-11-08 (window 2026-11-07, AWST; live reads 2026-11-08 00:0x-00:5x)
|
||||
|
||||
### The day
|
||||
|
||||
- **2026-11-07 settled 163 ledger rows — 159 `completed` / 3 `failed` / 1 `running`** (the *running* row is
|
||||
this audit, `cbfe434a3b5e`, fired 23:59:58). Same fleet as the previous five audits: **29 jobs, 26 enabled**.
|
||||
- **Deliveries: 7 rows / 7 `delivered` / 0 `for_failure`** (10 on each of 11-05 and 11-06 — no delivery fault
|
||||
to explain).
|
||||
|
||||
### The window's failure: a restart drain orphaned two in-flight runs
|
||||
|
||||
- Two of the three `failed` rows are **one event**. `ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 →
|
||||
05:02:22.870) and `1a8e90ea2271` (*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174) both closed with
|
||||
`RuntimeError: Cron job '…' lost its durable fire claim ownership`. The drain happened at **05:00:43/47**;
|
||||
systemd restarted the gateway at **05:01:00** (`Scheduled restart job, restart counter is at 3.`) after a
|
||||
requested restart sat in *after-turn* mode for the full **1800 s** (`#77184`).
|
||||
- The restart was requested at **~04:29:52** — 2 s before both *restart-safe cron workers* exited status 1
|
||||
(*"after adopting this execution, without writing a durable terminal state; whether side effects ran is
|
||||
unknown"*, logged 04:29:54) — i.e. **inside the `conduit_push` self-update window** (`0.122.2 → 0.134.4`,
|
||||
04:27:55 → 04:32:32, its own ledger recording `restarting_gateway` = **failed**). Working hypothesis, not
|
||||
proven: the plugin asked for the restart and could not perform it. The 0.22.5 → 0.23.0 updater is excluded
|
||||
(its receipt starts 05:05:13). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
||||
- The third `failed` row is **the watchdog working**: `96eb95ce12f2` (05:02:41.105 → 05:02:59.640) exited 1
|
||||
having caught the health monitor's `RuntimeError` traceback in `agent.log` — it reported the event ~19 s
|
||||
after it was written.
|
||||
|
||||
### Still open
|
||||
|
||||
- ⚠ **Unclosed incident rows 0 → 2, and neither alerted.** Both belong to the stocktake job and sit in state
|
||||
`detected` with **`alerted_at: null`**: `1a8e90_2f370ef764f5` (first seen 2026-11-07T05:02:21.573) and
|
||||
`1a8e90_628d914f1f97` (2026-11-07T04:29:54.815). The health monitor's two rows were closed 15:43:17 and
|
||||
resolved 17:08:52 the same day. This is the *red row, no alert* shape the 11-06 audit recorded for Job
|
||||
Radar, now reproduced on a different job.
|
||||
- ⚠ **Non-clean `last_status` 1 → 2:** `1a8e90ea2271` is now `error` with **`failure_streak: 2`** — the 03:00
|
||||
stocktake slot has failed on consecutive days (11-06 and 11-07) and again left its stocktake edits
|
||||
uncommitted; the 11-07 in-day entry folded those edits into its own commit so the tree was not left dirty.
|
||||
The other non-clean row is unchanged: `066bb5191280` (*Uber Eats Monthly Spend Report*), `delivery_failed`,
|
||||
streak 0, *"configured thread_id 34795 … was not found"*.
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] stays **OPEN** (untouched in-window).
|
||||
- **Job Radar ran clean on 11-07** — no slot failed, no claim conflict; the 11-06 concurrency page is
|
||||
unchanged.
|
||||
|
||||
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
|
||||
|
||||
- *Mnemosyne Health Monitor* `ea79a2107e36` — 5 h 01 m before the drain killed it (its two previous slots ran
|
||||
7 h 47 m 56 s and 12 h 13 m 54 s without losing the claim).
|
||||
- *wiki-stocktake-maintenance* `1a8e90ea2271` — 1 h 59 m 51 s before the drain killed it.
|
||||
|
||||
### Skill inventory movement (in-window — 2026-11-07 07:43 AWST)
|
||||
|
||||
- The stocktake's skills-index regeneration now reads **299 skills / 67 categories**, down from **302 / 68**,
|
||||
because **four skills were moved into `~/.hermes/skills/.archive/`** at **07:43**: `cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`. The parked archive grew **46 → 50**,
|
||||
and the `Cron Model Repair` category vanished with its only member.
|
||||
- ⚠ **No active skill references any of the four** — a grep of every `SKILL.md` outside `.archive/` returns
|
||||
no absorbing skill, so if this was a curator consolidation the *"absorbed into X"* pointer is missing.
|
||||
Their content survives only in the parked copies.
|
||||
- One skill **appeared**: `microsoft-office` (Productivity 25 → 26), part of a **10-file skill refresh at
|
||||
05:36** that lines up with the 0.22.5 → 0.23.0 update's own skill materialisation (the others were
|
||||
`computer-use`, `claude-design`, `popular-web-designs`, `document-to-action-items`, `meeting-action-items`,
|
||||
`product-price-monitor`, `weekly-review-planning`, `grounded-citations`, `dogfood`).
|
||||
- Filesystem vs. page: **300 `SKILL.md` on disk vs. 299 listed** — the generator de-dups one nested
|
||||
duplicate, so **299** is the unique figure (behaviour unchanged).
|
||||
|
||||
### Freshness / files touched
|
||||
|
||||
- **Freshness re-run:** `python3 raw/stocktake/_freshness_audit.py . raw/stocktake/freshness-20261108.txt 21`
|
||||
→ **0 stale / 126 fresh / 3 no-date of 129 pages** (report: `raw/stocktake/freshness-20261108.txt`).
|
||||
Nothing has crossed the 21-day line; the oldest pages are `CONTRIBUTING.md` and `agents/index.md` at 19 d.
|
||||
- **Pages touched by this audit:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
|
||||
[[systems/hermes-agent]] (version → 0.23.0, restart record),
|
||||
[[agents/hermes-production]] (version, 11-07 ledger), [[agents/hermes-chaos]] (CT450 stamp now 0.23.0),
|
||||
[[systems/browser-backend]] (camofox is now a plugin), [[systems/conduit]] (self-update 0.134.4, failed
|
||||
`restarting_gateway`), [[systems/hermes-hq-push]] (0.9.27 → 0.9.42), [[systems/opencode-go]] (stored key
|
||||
expired), [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (redaction-off standing
|
||||
condition), [[current-state]], [[index]], `log.md`, and the vault mirror (`/home/hermes/wiki`, both
|
||||
- **No wiki page removed or archived**, no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. The only archival action in-window was the four *skills* above, at the filesystem level.
|
||||
|
||||
Reference in New Issue
Block a user