wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts
This commit is contained in:
@@ -36,11 +36,11 @@ Primary agent memory store for Hermes.
|
||||
## Database Stats (live, 2026-10-04)
|
||||
| Table | Rows | Notes |
|
||||
|---|---|---|
|
||||
| `working_memory` | 752 | Active memories |
|
||||
| `episodic_memory` | 1,039 | Consolidated summaries |
|
||||
| `memory_embeddings` | 739 | Working-tier embeddings |
|
||||
| `working_memory` | 773 | Active memories |
|
||||
| `episodic_memory` | 1,047 | Consolidated summaries |
|
||||
| `memory_embeddings` | 760 | Working-tier embeddings |
|
||||
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||||
| `vec_working` / `vec_working_rowids` | 739 | Working vector index |
|
||||
| `vec_working` / `vec_working_rowids` | 760 | Working vector index |
|
||||
| `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index |
|
||||
| `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised |
|
||||
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||||
@@ -49,6 +49,8 @@ Primary agent memory store for Hermes.
|
||||
|
||||
## Known Issues
|
||||
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||||
|
||||
@@ -3,8 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-08
|
||||
verified_on: 2026-10-08
|
||||
updated: 2026-10-09
|
||||
verified_on: 2026-10-09
|
||||
review_after: 2026-10-23
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -118,3 +119,55 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
|
||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
|
||||
|
||||
---
|
||||
|
||||
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
|
||||
|
||||
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
|
||||
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
|
||||
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
|
||||
|
||||
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
|
||||
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
|
||||
every run from 18:00 onward completed. Full timeline, job list and incident table:
|
||||
[[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||||
|
||||
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
|
||||
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
|
||||
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
|
||||
03:00 run today decides whether the streak clears.
|
||||
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
|
||||
`agent.log`, so it reported the storm rather than suffering it.
|
||||
|
||||
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
|
||||
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
|
||||
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
|
||||
00:12). The 2026-10-09 run covers both days.
|
||||
|
||||
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
|
||||
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
|
||||
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
|
||||
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
|
||||
|
||||
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
|
||||
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
|
||||
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
|
||||
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
|
||||
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
|
||||
mutate cron state**; retiring the job is open item #1 below.
|
||||
|
||||
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
|
||||
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
|
||||
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
|
||||
See [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
### Open items (2026-10-09)
|
||||
|
||||
1. **Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.
|
||||
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
|
||||
3. **Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).
|
||||
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
|
||||
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
|
||||
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-07
|
||||
verified_on: 2026-10-07
|
||||
updated: 2026-10-08
|
||||
verified_on: 2026-10-08
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user