audit(2026-10-12): cron fleet clean 2nd day (10-11 171/0, 10-10 finalised 175/0); UTC-vs-AWST false 'capture stalled' alarm corrected; proxmox storage table + host mounts; queries/retained-research-answers stub filled (VNC); capture-pipeline incident -> superseded; index stale labels fixed
This commit is contained in:
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -51,6 +51,10 @@ Primary agent memory store for Hermes.
|
||||
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
**Tier counts (live, 2026-10-12 00:1x):** working `786`, episodic `1,056`, `facts` `876` (`fts_facts` 876 — FTS aligned), `fts_working` 786, `memoria_facts` `410`, `conflicts` `152`. Embeddings: working `773/786 (98.3%)`, episodic `1,052/1,056 (99.6%)` — both above the 95% reindex threshold; the coverage job `b63ed5817212` backfilled 15 vectors since the 2026-10-11 00:08 check.
|
||||
|
||||
**Capture clock is naive UTC — never compare it to AWST `now` (2026-10-11 false alarm).** At 2026-10-11 00:08 the health monitor raised an EXCEPTION claiming the capture path had stalled for 10.7 hours, citing `working_memory` `MAX(created_at) = 2026-10-10 13:24:53` against the local clock. `created_at` is stored as a **naive UTC** string (the same trap as `consolidation_log`), so the real gap was **2.7 h**: that write landed at 21:24:53 AWST, inside the 21:13 session that produced it. Capture was healthy throughout — `working_memory` gained 2 rows on 2026-10-11 (`11:19:37` and `11:51:39` UTC = 19:19 / 19:51 AWST), both from the day's only interactive session. Convert to local before judging staleness; a raw MAX-vs-now comparison always looks ~8 h stale in AWST.
|
||||
|
||||
### Vector embedding coverage gap — RESOLVED (2026-08-02)
|
||||
- Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories.
|
||||
- **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-12
|
||||
verified_on: 2026-10-12
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-06)
|
||||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, 2026-10-12; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
|
||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||||
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
||||
@@ -25,7 +25,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
|
||||
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
|
||||
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08)
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11)
|
||||
|
||||
## Job Table
|
||||
|
||||
@@ -214,3 +214,47 @@ the absence of job output rather than a positive lookup. Removals must be verifi
|
||||
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
|
||||
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
|
||||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||
|
||||
---
|
||||
|
||||
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
|
||||
|
||||
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
|
||||
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
|
||||
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
|
||||
|
||||
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
|
||||
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
|
||||
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
|
||||
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
|
||||
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
|
||||
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
|
||||
|
||||
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
|
||||
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
|
||||
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
|
||||
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
|
||||
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
|
||||
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
|
||||
|
||||
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
|
||||
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
|
||||
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
|
||||
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
|
||||
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
|
||||
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
|
||||
|
||||
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
|
||||
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
|
||||
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
|
||||
|
||||
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
|
||||
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
|
||||
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
|
||||
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
|
||||
`[SILENT]` (green). → [[systems/mnemosyne]]
|
||||
|
||||
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
|
||||
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
|
||||
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
|
||||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||||
|
||||
Reference in New Issue
Block a user