audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed

This commit is contained in:
Tony0410
2026-10-12 00:14:11 +08:00
parent d981e30128
commit 6fc109e546
8 changed files with 205 additions and 29 deletions
+6 -2
View File
@@ -3,8 +3,8 @@ title: Mnemosyne
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-12
verified_on: 2026-10-12
confidence: high
tags: [memory, mnemosyne, backend]
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
@@ -51,6 +51,10 @@ Primary agent memory store for Hermes.
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
### Vector embedding coverage gap — RESOLVED (2026-08-02)
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
+48 -4
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-12
verified_on: 2026-10-12
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
@@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06)
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, 2026-10-12; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
@@ -25,7 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08)
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11)
## Job Table
@@ -214,3 +214,47 @@ the absence of job output rather than a positive lookup. Removals must be verifi
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
---
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
`[SILENT]` (green). → [[systems/mnemosyne]]
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).