audit 2026-11-16: memory stall 62h54m (counts unchanged, cron_incidents 3->5 unclosed, consolidator zero-item streak 120); NEW incident — 11-15 plugin load-timeout storm (52 failed loads, 519 dropped registrations incl. conduit_push 474, 11 pre_tool_call + 2 pre_llm_call hook timeouts, 10 refusals) never recorded by the 11-15 audit; hook page Recurrence 2 (load average removed as predictor); fleet 29 jobs, 11-16 ledger 175 rows 173/2, deliveries 11/11
This commit is contained in:
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-16
|
||||
verified_on: 2026-11-16
|
||||
review_after: 2026-11-17
|
||||
updated: 2026-11-17
|
||||
verified_on: 2026-11-17
|
||||
review_after: 2026-11-18
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
|
||||
@@ -14,6 +14,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-17 00:27 (window 2026-11-16 00:00 -> 23:59; nightly audit).** Still **29
|
||||
jobs** (26 enabled, 3 disabled) — 11th consecutive window unchanged. 11-16 ledger: **175 rows — 173
|
||||
`completed`, 2 `failed`**; both failures are `ea79a2107e36` *Mnemosyne Health Monitor*
|
||||
(`failure_streak` **3 -> 5**). Deliveries **11/11 `delivered`**, 2 of them `for_failure` (that job's two
|
||||
alerts). 11-17 so far: 5 completed / 1 failed / 1 running (this audit). The 11-15 watchdog deps-guard
|
||||
**600 s timeout did not recur**. 11-16 also closed out the 11-15 saturation event: **0** plugin load
|
||||
failures and **0** hook timeouts, against 52 and 13 the day before — see
|
||||
[[incidents/2026-11-15-plugin-load-timeout-storm]].
|
||||
|
||||
- **Current as of 2026-11-15 (window 2026-11-14 23:59 -> 2026-11-15 23:59; nightly audit).** **168 rows — 165 `completed` / 3 `failed`**, deliveries **9 / 9 `delivered`** (**3** `for_failure`). The three failures, in order: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604` exit 1; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` — **17 m 49.8 s** — exit 1 with `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds` (**a new signature for that job**; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430` exit 1. **2026-11-14 settled 167 / 167 `completed` / 0 `failed`** (0 `running` — that audit finished, so nothing was silently in flight). Fleet **29 jobs (26 enabled / 3 disabled)** — a **tenth consecutive window at the same count**; the only non-clean `last_status` remains `ea79a2107e36` (`error`, `failure_streak` **3**). `cron_incidents` **1 -> 3 unclosed** (**141** rows) — all three this job, all `failure_type: auth`, each `first_seen_at == last_seen_at`; the signature embeds the elapsed-hours float so the same fault fragments a new row per slot (full analysis on [[incidents/2026-11-14-mnemosyne-capture-stall]]). Sessions started 11-15: **9, all `source: cron`** — no interactive session. 2026-11-16 so far (live 00:2x): 7 rows — 5 `completed` / 1 `failed` (the monitor's `00:01:33.030` slot) / 1 `running` (this audit).
|
||||
- **Current as of 2026-11-14 (window 2026-11-13 23:59 -> 2026-11-14 23:59; nightly audit).** **A fourth
|
||||
consecutive zero-`failed` day for the fleet — then the memory monitor broke the clean streak at 00:03.**
|
||||
|
||||
Reference in New Issue
Block a user