wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-08
|
||||
> model routing and automation. Last updated: 2026-10-09
|
||||
|
||||
## Start here
|
||||
- [[current-state]] — Current systems status
|
||||
@@ -28,10 +28,12 @@
|
||||
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **0 failing as of 2026-10-08** — 7 dead-model-pin jobs repinned 2026-10-05, 4 now verified by live run; one transient Camofox timeout 2026-10-07 auto-resolved)
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]])
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **10 carry a non-clean `last_status` as of 2026-10-10 00:22** — 9 from the 2026-10-09 fleet-wide provider timeout storm (15 failed executions, 10 jobs) plus 1 `delivery_failed`; the 7 dead-model-pin jobs repinned 2026-10-05 stay healthy. See [[incidents/2026-10-09-cron-provider-timeout-storm]])
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]])
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 8 incidents still open (unresolved)
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
|
||||
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
||||
|
||||
Reference in New Issue
Block a user