wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts
This commit is contained in:
+10
-2
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-08
|
||||
verified_on: 2026-10-08
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -14,6 +14,14 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
||||
# Current State
|
||||
Updated: 2026-10-08
|
||||
|
||||
## Recent Changes (2026-10-09)
|
||||
|
||||
- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||||
- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]]
|
||||
- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]].
|
||||
- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]]
|
||||
|
||||
## Recent Changes (2026-10-08)
|
||||
- **Cron fleet green for a third consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters still read 0. Window tally (`executions.db`, 2026-10-07 00:00 → 2026-10-08 00:00 AWST): **176 `completed`, 1 `failed`, 1 `running`** (the nightly audit itself). 13 sessions started in the window, all `source = cron` — **no interactive (non-cron) sessions**; 12 deliveries, **0 delivery failures**.
|
||||
- **First detect → notify → auto-resolve incident cycle since the managed-runtime event (DHL, 2026-10-07 08:00 → 11:01).** `62619aa6bcb3` (DHL Tracking, 5×/day) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` — Camofox briefly unresponsive on a browser HTTP call (`dhl_tracking_monitor.py:233`). The failure notice went out at 08:00:58 and incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) **auto-resolved at 11:01:13** when the next 3-hourly run completed; the 14:00, 17:00 and 20:00 runs also succeeded. No action needed — and the incident machinery behaved exactly as designed, which it had not been tested on since 2026-09-30. See [[systems/scheduled-tasks]].
|
||||
|
||||
Reference in New Issue
Block a user