wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts

This commit is contained in:
Tony0410
2026-10-10 00:27:15 +08:00
parent 659356aa61
commit 8cab69bbd0
9 changed files with 418 additions and 13 deletions
+6 -4
View File
@@ -36,11 +36,11 @@ Primary agent memory store for Hermes.
## Database Stats (live, 2026-10-04)
| Table | Rows | Notes |
|---|---|---|
| `working_memory` | 752 | Active memories |
| `episodic_memory` | 1,039 | Consolidated summaries |
| `memory_embeddings` | 739 | Working-tier embeddings |
| `working_memory` | 773 | Active memories |
| `episodic_memory` | 1,047 | Consolidated summaries |
| `memory_embeddings` | 760 | Working-tier embeddings |
| `memory` (legacy) | 2,181 | Old table, not migrated |
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
| `vec_working` / `vec_working_rowids` | 760 | Working vector index |
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
| Shared surface | 0 | Empty — cross-agent sharing not functional |
@@ -49,6 +49,8 @@ Primary agent memory store for Hermes.
## Known Issues
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
+55 -2
View File
@@ -3,8 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-08
verified_on: 2026-10-08
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-23
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -118,3 +119,55 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
---
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
[[incidents/2026-10-09-cron-provider-timeout-storm]].
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
03:00 run today decides whether the streak clears.
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
`agent.log`, so it reported the storm rather than suffering it.
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
00:12). The 2026-10-09 run covers both days.
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
mutate cron state**; retiring the job is open item #1 below.
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
See [[incidents/2026-10-09-memory-recall-relevance]].
### Open items (2026-10-09)
1. **Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
3. **Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-10-07
verified_on: 2026-10-07
updated: 2026-10-08
verified_on: 2026-10-08
confidence: high
tags: [skills, hermes]
sources: []