wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts

This commit is contained in:
Tony0410
2026-10-10 00:27:15 +08:00
parent 659356aa61
commit 8cab69bbd0
9 changed files with 418 additions and 13 deletions
@@ -0,0 +1,131 @@
---
title: Fleet-Wide Provider Timeout Storm — 2026-10-09
type: incident
status: unresolved
created: 2026-10-09
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-16
confidence: high
tags: [cron, omniroute, router, timeout, incident, unresolved]
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
---
# Fleet-Wide Provider Timeout Storm — 2026-10-09
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
**Status:** Unresolved — the storm stopped and four incidents auto-closed, but **eight incidents opened on 2026-10-08/09 are still open** and `1a8e90ea2271` is the only job left at `failure_streak: 2`
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
## What happened
Every failure in the window is the same shape — `RuntimeError: Request timed out.` — i.e. the Hermes
runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases
(`smart-route`, `free-stack`, `deepseek-paid-smart`) on provider `custom:omni-101`, whose base URL is
the local OmniRoute gateway. A job pinned to its own provider+model does **not** fall back to the
global `fallback_providers` chain, which is why ten unrelated jobs all failed at once instead of
quietly rerouting (see [[incidents/2026-10-05-dead-model-pin-repin]]).
## Timeline (2026-10-09 AWST)
| Time | Job | Error |
|------|-----|-------|
| 00:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. |
| 00:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 03:00 | `1a8e90ea2271` (wiki-stocktake-maintenance) | RuntimeError: Request timed out. |
| 06:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 06:30 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 |
| 06:45 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 |
| 07:00 | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | RuntimeError: Request timed out. |
| 07:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. |
| 07:00 | `dca8482e4f76` (Daily Marketing Job Radar) | RuntimeError: Request timed out. |
| 08:00 | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | RuntimeError: Request timed out. |
| 08:30 | `08eb86f02d37` (Gmail Action Digest) | RuntimeError: Request timed out. |
| 12:00 | `ecc1fc13655a` (FreshRSS Midday News Briefing) | RuntimeError: Request timed out. |
| 12:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. |
| 12:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. |
| 15:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. |
| 23:59 | `cbfe434a3b5e` (Wiki Audit — Nightly) | (no error text recorded) |
The 16 failed executions cover 11 distinct jobs:
| Job | Failures | Note |
|-----|----------|------|
| `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | 3 | provider timeout |
| `ea79a2107e36` (Mnemosyne Health Monitor) | 2 | provider timeout |
| `96eb95ce12f2` (Mnemosyne Watchdog) | 2 | cascade — watchdog, not victim |
| `1b31187397a5` (Mosquito Tracker) | 2 | provider timeout |
| `1a8e90ea2271` (wiki-stocktake-maintenance) | 1 | provider timeout |
| `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | 1 | provider timeout |
| `dca8482e4f76` (Daily Marketing Job Radar) | 1 | provider timeout |
| `1b23c042e17d` (Enhanced Weekend Adventure Planner) | 1 | provider timeout |
| `08eb86f02d37` (Gmail Action Digest) | 1 | provider timeout |
| `ecc1fc13655a` (FreshRSS Midday News Briefing) | 1 | provider timeout |
| `cbfe434a3b5e` (Wiki Audit — Nightly) | 1 | provider timeout |
**Not established:** whether the trigger was OmniRoute saturation, an upstream provider outage, or
the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias,
not from inside the storm window) failed to connect outright (`HTTP 000`) — corroborating
intermittent reachability, but recorded here as an observation, not a root cause.
## Recovery
Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run:
`96eb95_acb3ae5ec223` 07:01, `b63ed5_f2c93dd24181` 18:01, `1b3118_f49c8d2c6e7f` 22:05, and
`ea79a2_6fb70b032305` at **2026-10-10 00:03:44** (the first post-window run). The gateway was
therefore serving normally again well before the window closed.
## Incidents
| Incident | Job | State | Failure type | First seen | Closed |
|----------|-----|-------|--------------|------------|--------|
| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | unknown | 2026-10-08 03:01 | — |
| `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 |
| `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 |
| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | detected | timeout | 2026-10-09 00:32 | — |
| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | timeout | 2026-10-09 03:17 | — |
| `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 |
| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | alerted | timeout | 2026-10-09 07:18 | — |
| `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 |
| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | alerted | timeout | 2026-10-09 07:18 | — |
| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | alerted | timeout | 2026-10-09 08:17 | — |
| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | alerted | timeout | 2026-10-09 08:48 | — |
| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | alerted | timeout | 2026-10-09 12:21 | — |
## Why it reached Anthony
`b63ed5817212` (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00)
while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not
fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is
where the DHL-monitor retirement was requested (see [[systems/scheduled-tasks]], open item #1).
## The watchdog cascade
`96eb95ce12f2` (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with
`Script exited with code 1 — 🚨 Log ERROR in agent.log`, which is the watchdog **reporting** the storm
rather than suffering it. Read its failures as a symptom, not as two more victims.
## Evidence (how to re-check)
```bash
sqlite3 ~/.hermes/cron/executions.db \
"SELECT status,count(*) FROM executions
WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1"
# today's read: completed 162 | failed 15 | running 1
sqlite3 ~/.hermes/cron/executions.db \
"SELECT job_id,started_at,substr(error,1,60) FROM executions
WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'"
sqlite3 ~/.hermes/cron/executions.db \
"SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL"
```
Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}.
## Related
- [[systems/scheduled-tasks]] — the job register and this window's re-verification block
- [[incidents/2026-10-05-dead-model-pin-repin]] — why the fleet is aliased on `custom:omni-101`
- [[concepts/model-routing]] — routing aliases and fallback semantics
- [[systems/mnemosyne]] — the three memory jobs caught in the storm
@@ -0,0 +1,81 @@
---
title: Memory Recall Relevance — a Stale Episode Outranked Current Facts
type: incident
status: unresolved
created: 2026-10-09
updated: 2026-10-09
verified_on: 2026-10-09
review_after: 2026-10-16
confidence: medium
tags: [memory, mnemosyne, recall, incident, unresolved]
sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml]
---
# Memory Recall Relevance — a Stale Episode Outranked Current Facts
**Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile)
**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported
saved and is **not** present in live config.
## Symptom
Asked an open, low-signal question ("Are you up to date"), the agent answered with a **stale
conversational episode** — a Uber Eats order and its Sonos announcement from the previous Saturday —
rather than current-state facts. Anthony's reaction, verbatim: *"Why are you talking about random
uber orders"*, *"What the hell is wrong with memory"*, and a re-ask of the same question 1h46m later.
## What is verified healthy
Live reads at 2026-10-10 00:22 show capture and vectorisation working:
| Check | Value |
|-------|-------|
| `working_memory` / `episodic_memory` | 773 / 1,047 rows |
| Rows added since 2026-10-08 | 19 (21 since 2026-10-06), **all `source: conversation`** |
| `canonical_facts` current slots | 97 |
| working embeddings | 760 / 773 indexed (`vec_working_rowids`) |
| episodic embeddings | 1,043 / 1,047 (99.6%) |
| DB mtime | 2026-10-09 22:27 |
So this was **not** the 2026-10-03-style failure where embeddings were missing and recall fell back
to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]).
## The ranking problem
- **Importance is a weak discriminator.** All 1,047 episodic rows carry exactly `0.6`; 653 of 773
working rows carry `0.5`, and only ~99 rows sit above `0.8`. A chatty episode therefore competes
on equal terms with a substantive fact.
- **Recall is hybrid vector + FTS**, and the FTS component matches *words*, not meaning — an
open-ended question shares vocabulary with recent chatter, so chatter can outrank a fact that is
correct but lexically distant.
- **Canonical facts sit on a separate path.** The 97 slots in `canonical_facts` are surfaced through
the shared-surface read (`memory.mnemosyne.shared_surface_read: true`,
`shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection
and ranking, not a switched-off injection path.
## What did not get fixed
The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your
profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22:
- **no top-level `mnemosyne:` key**, and
- `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`,
`importance_weight` are **absent**.
Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the
fix as not applied until a weight change or a reranking strategy is actually verified in config.
## Open items
1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit
profile/canonical-first rule) — then re-run the same question as a regression test.
2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a
table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes).
3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that
should decay, or the writer is not distinguishing signal at capture time.
## Related
- [[systems/mnemosyne]] — live tier counts and coverage
- [[concepts/agent-memory]] — memory model
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — the earlier, different recall failure