wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts

This commit is contained in:
Tony0410
2026-10-10 00:27:15 +08:00
parent 659356aa61
commit 8cab69bbd0
9 changed files with 418 additions and 13 deletions
+55 -2
View File
@@ -3,8 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-08
verified_on: 2026-10-08
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-23
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -118,3 +119,55 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
---
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
[[incidents/2026-10-09-cron-provider-timeout-storm]].
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
03:00 run today decides whether the streak clears.
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
`agent.log`, so it reported the storm rather than suffering it.
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
00:12). The 2026-10-09 run covers both days.
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
mutate cron state**; retiring the job is open item #1 below.
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
See [[incidents/2026-10-09-memory-recall-relevance]].
### Open items (2026-10-09)
1. **Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
3. **Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.