wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts

This commit is contained in:
Tony0410
2026-10-10 00:27:15 +08:00
parent 659356aa61
commit 8cab69bbd0
9 changed files with 418 additions and 13 deletions
+25
View File
@@ -395,3 +395,28 @@ Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.
**Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
**Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and
produced no entry, so this run covers two days.
**Pages written**
- `incidents/2026-10-09-cron-provider-timeout-storm.md` — new.
- `incidents/2026-10-09-memory-recall-relevance.md` — new.
- `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
- `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
- `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped.
- `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
- `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed.
**Findings worth the ink**
- **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it.
- **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing.
- **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`.
- **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
- **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key).
**Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.