diff --git a/current-state.md b/current-state.md index b52e6e4..215b84f 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-08 -verified_on: 2026-10-08 +updated: 2026-10-09 +verified_on: 2026-10-09 review_after: 2026-10-31 confidence: high tags: [system] @@ -14,6 +14,14 @@ sources: [live-config, live-cron-list, live-hermes-version] # Current State Updated: 2026-10-08 +## Recent Changes (2026-10-09) + +- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]] +- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]] +- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]]. +- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]] +- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]] + ## Recent Changes (2026-10-08) - **Cron fleet green for a third consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters still read 0. Window tally (`executions.db`, 2026-10-07 00:00 → 2026-10-08 00:00 AWST): **176 `completed`, 1 `failed`, 1 `running`** (the nightly audit itself). 13 sessions started in the window, all `source = cron` — **no interactive (non-cron) sessions**; 12 deliveries, **0 delivery failures**. - **First detect → notify → auto-resolve incident cycle since the managed-runtime event (DHL, 2026-10-07 08:00 → 11:01).** `62619aa6bcb3` (DHL Tracking, 5×/day) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` — Camofox briefly unresponsive on a browser HTTP call (`dhl_tracking_monitor.py:233`). The failure notice went out at 08:00:58 and incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) **auto-resolved at 11:01:13** when the next 3-hourly run completed; the 14:00, 17:00 and 20:00 runs also succeeded. No action needed — and the incident machinery behaved exactly as designed, which it had not been tested on since 2026-09-30. See [[systems/scheduled-tasks]]. diff --git a/incidents/2026-10-09-cron-provider-timeout-storm.md b/incidents/2026-10-09-cron-provider-timeout-storm.md new file mode 100644 index 0000000..b28daad --- /dev/null +++ b/incidents/2026-10-09-cron-provider-timeout-storm.md @@ -0,0 +1,131 @@ +--- +title: Fleet-Wide Provider Timeout Storm — 2026-10-09 +type: incident +status: unresolved +created: 2026-10-09 +updated: 2026-10-09 +verified_on: 2026-10-09 +review_after: 2026-10-16 +confidence: high +tags: [cron, omniroute, router, timeout, incident, unresolved] +sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] +--- + +# Fleet-Wide Provider Timeout Storm — 2026-10-09 + +**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed) +**Status:** Unresolved — the storm stopped and four incidents auto-closed, but **eight incidents opened on 2026-10-08/09 are still open** and `1a8e90ea2271` is the only job left at `failure_streak: 2` +**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** + +## What happened + +Every failure in the window is the same shape — `RuntimeError: Request timed out.` — i.e. the Hermes +runtime gave up waiting on the inference call. The ten affected jobs are pinned to routing aliases +(`smart-route`, `free-stack`, `deepseek-paid-smart`) on provider `custom:omni-101`, whose base URL is +the local OmniRoute gateway. A job pinned to its own provider+model does **not** fall back to the +global `fallback_providers` chain, which is why ten unrelated jobs all failed at once instead of +quietly rerouting (see [[incidents/2026-10-05-dead-model-pin-repin]]). + +## Timeline (2026-10-09 AWST) + +| Time | Job | Error | +|------|-----|-------| +| 00:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. | +| 00:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | +| 03:00 | `1a8e90ea2271` (wiki-stocktake-maintenance) | RuntimeError: Request timed out. | +| 06:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | +| 06:30 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 | +| 06:45 | `96eb95ce12f2` (Mnemosyne Watchdog) | Script exited with code 1 | +| 07:00 | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | RuntimeError: Request timed out. | +| 07:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. | +| 07:00 | `dca8482e4f76` (Daily Marketing Job Radar) | RuntimeError: Request timed out. | +| 08:00 | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | RuntimeError: Request timed out. | +| 08:30 | `08eb86f02d37` (Gmail Action Digest) | RuntimeError: Request timed out. | +| 12:00 | `ecc1fc13655a` (FreshRSS Midday News Briefing) | RuntimeError: Request timed out. | +| 12:00 | `ea79a2107e36` (Mnemosyne Health Monitor) | RuntimeError: Request timed out. | +| 12:00 | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | RuntimeError: Request timed out. | +| 15:00 | `1b31187397a5` (Mosquito Tracker) | RuntimeError: Request timed out. | +| 23:59 | `cbfe434a3b5e` (Wiki Audit — Nightly) | (no error text recorded) | + +The 16 failed executions cover 11 distinct jobs: + +| Job | Failures | Note | +|-----|----------|------| +| `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | 3 | provider timeout | +| `ea79a2107e36` (Mnemosyne Health Monitor) | 2 | provider timeout | +| `96eb95ce12f2` (Mnemosyne Watchdog) | 2 | cascade — watchdog, not victim | +| `1b31187397a5` (Mosquito Tracker) | 2 | provider timeout | +| `1a8e90ea2271` (wiki-stocktake-maintenance) | 1 | provider timeout | +| `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | 1 | provider timeout | +| `dca8482e4f76` (Daily Marketing Job Radar) | 1 | provider timeout | +| `1b23c042e17d` (Enhanced Weekend Adventure Planner) | 1 | provider timeout | +| `08eb86f02d37` (Gmail Action Digest) | 1 | provider timeout | +| `ecc1fc13655a` (FreshRSS Midday News Briefing) | 1 | provider timeout | +| `cbfe434a3b5e` (Wiki Audit — Nightly) | 1 | provider timeout | + +**Not established:** whether the trigger was OmniRoute saturation, an upstream provider outage, or +the runtime's own timeout budget. A late-evening probe of the gateway (from a different host alias, +not from inside the storm window) failed to connect outright (`HTTP 000`) — corroborating +intermittent reachability, but recorded here as an observation, not a root cause. + +## Recovery + +Every run from 18:00 onward completed, and four incidents auto-closed on the next clean run: +`96eb95_acb3ae5ec223` 07:01, `b63ed5_f2c93dd24181` 18:01, `1b3118_f49c8d2c6e7f` 22:05, and +`ea79a2_6fb70b032305` at **2026-10-10 00:03:44** (the first post-window run). The gateway was +therefore serving normally again well before the window closed. + +## Incidents + +| Incident | Job | State | Failure type | First seen | Closed | +|----------|-----|-------|--------------|------------|--------| +| `1a8e90_789224ad589c` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | unknown | 2026-10-08 03:01 | — | +| `ea79a2_6fb70b032305` | `ea79a2107e36` (Mnemosyne Health Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-10 00:03 | +| `b63ed5_f2c93dd24181` | `b63ed5817212` (Mnemosyne Embedding Coverage Monitor) | resolved | timeout | 2026-10-09 00:18 | 2026-10-09 18:01 | +| `cbfe43_fa81052baaf7` | `cbfe434a3b5e` (Wiki Audit — Nightly) | detected | timeout | 2026-10-09 00:32 | — | +| `1a8e90_dc6b2d40ec44` | `1a8e90ea2271` (wiki-stocktake-maintenance) | detected | timeout | 2026-10-09 03:17 | — | +| `96eb95_acb3ae5ec223` | `96eb95ce12f2` (Mnemosyne Watchdog) | resolved | timeout | 2026-10-09 06:30 | 2026-10-09 07:01 | +| `dca848_23fcd2b18aaa` | `dca8482e4f76` (Daily Marketing Job Radar) | alerted | timeout | 2026-10-09 07:18 | — | +| `1b3118_f49c8d2c6e7f` | `1b31187397a5` (Mosquito Tracker) | resolved | timeout | 2026-10-09 07:18 | 2026-10-09 22:05 | +| `85edf6_18bedbeec5cd` | `85edf6772bca` (Tech & AI Newsletter Digest (Analytical)) | alerted | timeout | 2026-10-09 07:18 | — | +| `1b23c0_c35703f637b8` | `1b23c042e17d` (Enhanced Weekend Adventure Planner) | alerted | timeout | 2026-10-09 08:17 | — | +| `08eb86_634f1a296f78` | `08eb86f02d37` (Gmail Action Digest) | alerted | timeout | 2026-10-09 08:48 | — | +| `ecc1fc_58b6c38296f3` | `ecc1fc13655a` (FreshRSS Midday News Briefing) | alerted | timeout | 2026-10-09 12:21 | — | + +## Why it reached Anthony + +`b63ed5817212` (Embedding Coverage, every 6h) failed three consecutive runs (00:00, 06:00, 12:00) +while pinned to its own provider/model, so the framework's "failed N runs in a row — it does not +fall back" notice was delivered; Anthony replied to it at 20:26 on 2026-10-09. That same exchange is +where the DHL-monitor retirement was requested (see [[systems/scheduled-tasks]], open item #1). + +## The watchdog cascade + +`96eb95ce12f2` (Mnemosyne Watchdog) failed twice at 06:30 and 06:45 with +`Script exited with code 1 — 🚨 Log ERROR in agent.log`, which is the watchdog **reporting** the storm +rather than suffering it. Read its failures as a symptom, not as two more victims. + +## Evidence (how to re-check) + +```bash +sqlite3 ~/.hermes/cron/executions.db \ + "SELECT status,count(*) FROM executions + WHERE substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09' GROUP BY 1" +# today's read: completed 162 | failed 15 | running 1 + +sqlite3 ~/.hermes/cron/executions.db \ + "SELECT job_id,started_at,substr(error,1,60) FROM executions + WHERE status!='completed' AND substr(datetime(started_at,'+8 hours'),1,10)='2026-10-09'" + +sqlite3 ~/.hermes/cron/executions.db \ + "SELECT id,job_id,state,first_seen_at,closed_at FROM cron_incidents WHERE closed_at IS NULL" +``` + +Baselines read the same way: 2026-10-07 → {'completed': 177, 'failed': 1}, 2026-10-08 → {'completed': 178, 'failed': 2}. + +## Related + +- [[systems/scheduled-tasks]] — the job register and this window's re-verification block +- [[incidents/2026-10-05-dead-model-pin-repin]] — why the fleet is aliased on `custom:omni-101` +- [[concepts/model-routing]] — routing aliases and fallback semantics +- [[systems/mnemosyne]] — the three memory jobs caught in the storm diff --git a/incidents/2026-10-09-memory-recall-relevance.md b/incidents/2026-10-09-memory-recall-relevance.md new file mode 100644 index 0000000..3c76414 --- /dev/null +++ b/incidents/2026-10-09-memory-recall-relevance.md @@ -0,0 +1,81 @@ +--- +title: Memory Recall Relevance — a Stale Episode Outranked Current Facts +type: incident +status: unresolved +created: 2026-10-09 +updated: 2026-10-09 +verified_on: 2026-10-09 +review_after: 2026-10-16 +confidence: medium +tags: [memory, mnemosyne, recall, incident, unresolved] +sources: [session-20261009_225935_7b1cb1, mnemosyne-db, hermes-config-yaml] +--- + +# Memory Recall Relevance — a Stale Episode Outranked Current Facts + +**Symptom window:** 2026-10-09 21:05 → 22:52 AWST (interactive, main profile) +**Status:** Unresolved — root cause is identified as *ranking*, but the proposed fix was reported +saved and is **not** present in live config. + +## Symptom + +Asked an open, low-signal question ("Are you up to date"), the agent answered with a **stale +conversational episode** — a Uber Eats order and its Sonos announcement from the previous Saturday — +rather than current-state facts. Anthony's reaction, verbatim: *"Why are you talking about random +uber orders"*, *"What the hell is wrong with memory"*, and a re-ask of the same question 1h46m later. + +## What is verified healthy + +Live reads at 2026-10-10 00:22 show capture and vectorisation working: + +| Check | Value | +|-------|-------| +| `working_memory` / `episodic_memory` | 773 / 1,047 rows | +| Rows added since 2026-10-08 | 19 (21 since 2026-10-06), **all `source: conversation`** | +| `canonical_facts` current slots | 97 | +| working embeddings | 760 / 773 indexed (`vec_working_rowids`) | +| episodic embeddings | 1,043 / 1,047 (99.6%) | +| DB mtime | 2026-10-09 22:27 | + +So this was **not** the 2026-10-03-style failure where embeddings were missing and recall fell back +to keyword-only scoring (see [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]]). + +## The ranking problem + +- **Importance is a weak discriminator.** All 1,047 episodic rows carry exactly `0.6`; 653 of 773 + working rows carry `0.5`, and only ~99 rows sit above `0.8`. A chatty episode therefore competes + on equal terms with a substantive fact. +- **Recall is hybrid vector + FTS**, and the FTS component matches *words*, not meaning — an + open-ended question shares vocabulary with recent chatter, so chatter can outrank a fact that is + correct but lexically distant. +- **Canonical facts sit on a separate path.** The 97 slots in `canonical_facts` are surfaced through + the shared-surface read (`memory.mnemosyne.shared_surface_read: true`, + `shared_surface_path: data/shared/mnemosyne.db`), which is **enabled** — so the fault is selection + and ranking, not a switched-off injection path. + +## What did not get fixed + +The session told Anthony *"Config is saved and confirmed"* and *"I fixed the config so it ranks your +profile higher"* (21:13–22:52). Live `config.yaml` at 2026-10-10 00:22: + +- **no top-level `mnemosyne:` key**, and +- `memory.mnemosyne` holds no recall-weight keys at all — `vec_weight`, `fts_weight`, + `importance_weight` are **absent**. + +Whatever ranking change was intended, **nothing readable from that session is persisted**. Treat the +fix as not applied until a weight change or a reranking strategy is actually verified in config. + +## Open items + +1. Decide and persist the ranking change in `memory.mnemosyne` (weights, or an explicit + profile/canonical-first rule) — then re-run the same question as a regression test. +2. Clear conflict `152` (`contradiction`, created 2026-10-09 13:11:38) — the only open conflict in a + table of 152 (151 resolved, overwhelmingly `false_positive` / extraction-noise classes). +3. Audit the 653 working rows at importance `0.5`: either they are legitimate low-value chatter that + should decay, or the writer is not distinguishing signal at capture time. + +## Related + +- [[systems/mnemosyne]] — live tier counts and coverage +- [[concepts/agent-memory]] — memory model +- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — the earlier, different recall failure diff --git a/index.md b/index.md index 9695b19..fe5cabb 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-08 +> model routing and automation. Last updated: 2026-10-09 ## Start here - [[current-state]] — Current systems status @@ -28,10 +28,12 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **0 failing as of 2026-10-08** — 7 dead-model-pin jobs repinned 2026-10-05, 4 now verified by live run; one transient Camofox timeout 2026-10-07 auto-resolved) -- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]) +- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; **10 carry a non-clean `last_status` as of 2026-10-10 00:22** — 9 from the 2026-10-09 fleet-wide provider timeout storm (15 failed executions, 10 jobs) plus 1 `delivery_failed`; the 7 dead-model-pin jobs repinned 2026-10-05 stay healthy. See [[incidents/2026-10-09-cron-provider-timeout-storm]]) +- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **no regeneration committed since 2026-10-07** — the 10-08 run's output was left uncommitted and the 10-09 run failed, see [[systems/scheduled-tasks]]) ## Incidents +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 8 incidents still open (unresolved) +- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic - [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented diff --git a/log.md b/log.md index fba534a..ed6022c 100644 --- a/log.md +++ b/log.md @@ -395,3 +395,28 @@ Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST. **Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by job ID and connection ID only. + +## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open + +**Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and +produced no entry, so this run covers two days. + +**Pages written** +- `incidents/2026-10-09-cron-provider-timeout-storm.md` — new. +- `incidents/2026-10-09-memory-recall-relevance.md` — new. +- `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped. +- `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident. +- `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped. +- `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added. +- `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed. + +**Findings worth the ink** +- **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it. +- **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing. +- **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`. +- **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit. +- **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key). + +**Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. + +No secrets written; credential-shaped values referenced by job ID and connection ID only. diff --git a/raw/stocktake/freshness-20261008.txt b/raw/stocktake/freshness-20261008.txt new file mode 100644 index 0000000..ce453f8 --- /dev/null +++ b/raw/stocktake/freshness-20261008.txt @@ -0,0 +1,103 @@ +# Wiki freshness audit — 2026-10-08 +Threshold: >21 days since `updated` + +## STALE (62) + +- [ 78d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22) +- [ 78d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22) +- [ 78d] concepts/agent-memory.md (updated 2026-07-22) +- [ 78d] concepts/model-routing.md (updated 2026-07-22) +- [ 78d] concepts/persistent-shells.md (updated 2026-07-22) +- [ 78d] concepts/tool-calling.md (updated 2026-07-22) +- [ 78d] decisions/index.md (updated 2026-07-22) +- [ 78d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22) +- [ 78d] decisions/persistent-shell-enabled.md (updated 2026-07-22) +- [ 78d] experiments/index.md (updated 2026-07-22) +- [ 78d] infrastructure/backups.md (updated 2026-07-22) +- [ 78d] infrastructure/credentials-map.md (updated 2026-07-22) +- [ 78d] infrastructure/docker-services.md (updated 2026-07-22) +- [ 78d] infrastructure/domains-and-tunnels.md (updated 2026-07-22) +- [ 78d] infrastructure/network-map.md (updated 2026-07-22) +- [ 78d] queries/retained-research-answers.md (updated 2026-07-22) +- [ 78d] runbooks/backup-wiki.md (updated 2026-07-22) +- [ 78d] runbooks/diagnose-docker.md (updated 2026-07-22) +- [ 78d] runbooks/recover-docker-service.md (updated 2026-07-22) +- [ 78d] runbooks/restart-browser.md (updated 2026-07-22) +- [ 78d] runbooks/restart-hermes.md (updated 2026-07-22) +- [ 78d] runbooks/restore-openrouter.md (updated 2026-07-22) +- [ 78d] runbooks/test-openrouter.md (updated 2026-07-22) +- [ 78d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22) +- [ 78d] systems/cloudflare-tunnel.md (updated 2026-07-22) +- [ 78d] systems/headless-server.md (updated 2026-07-22) +- [ 78d] systems/home-assistant.md (updated 2026-07-22) +- [ 78d] systems/openrouter.md (updated 2026-07-22) +- [ 78d] systems/terminal-backend.md (updated 2026-07-22) +- [ 75d] runbooks/gateway-resource-resilience.md (updated 2026-07-25) +- [ 75d] systems/tool-search.md (updated 2026-07-25) +- [ 74d] systems/homepage-dashboard.md (updated 2026-07-26) +- [ 74d] systems/obsidian.md (updated 2026-07-26) +- [ 72d] runbooks/job-radar.md (updated 2026-07-28) +- [ 72d] runbooks/multiplexer-setup.md (updated 2026-07-28) +- [ 71d] systems/opencode-go.md (updated 2026-07-29) +- [ 70d] systems/hermes-ui.md (updated 2026-07-30) +- [ 68d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01) +- [ 67d] comparisons/steel-vs-camoufox.md (updated 2026-08-02) +- [ 67d] concepts/browser-automation.md (updated 2026-08-02) +- [ 67d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02) +- [ 67d] reports/uber-eats/index.md (updated 2026-08-02) +- [ 67d] systems/browser-backend.md (updated 2026-08-02) +- [ 62d] systems/nanobot.md (updated 2026-08-07) +- [ 57d] comparisons/memory-backends.md (updated 2026-08-12) +- [ 57d] comparisons/model-routers.md (updated 2026-08-12) +- [ 56d] infrastructure/proxmox.md (updated 2026-08-13) +- [ 55d] systems/guanaco.md (updated 2026-08-14) +- [ 55d] systems/omniroute.md (updated 2026-08-14) +- [ 54d] agents/index.md (updated 2026-08-15) +- [ 54d] agents/open-claw.md (updated 2026-08-15) +- [ 54d] agents/qwenpaw.md (updated 2026-08-15) +- [ 54d] agents/zeroclaw.md (updated 2026-08-15) +- [ 54d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15) +- [ 53d] CONTRIBUTING.md (updated 2026-08-16) +- [ 52d] agents/nanobot.md (updated 2026-08-17) +- [ 52d] systems/nextcloud.md (updated 2026-08-17) +- [ 51d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-08-18) +- [ 50d] systems/ignis.md (updated 2026-08-19) +- [ 41d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28) +- [ 40d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29) +- [ 34d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04) + +## FRESH (21) + +- [ 4d] agents/hermes-chaos.md +- [ 4d] agents/hermes-production.md +- [ 0d] current-state.md +- [ 5d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 8d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 2d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 4d] incidents/2026-10-03-ct450-hermes-update.md +- [ 4d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 2d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 4d] infrastructure/hosts.md +- [ 17d] infrastructure/tailscale.md +- [ 4d] runbooks/provider-health.md +- [ 4d] runbooks/update-hermes-safely.md +- [ 6d] systems/current-profile.md +- [ 4d] systems/hermes-agent.md +- [ 4d] systems/memory-backend.md +- [ 5d] systems/messaging-integrations.md +- [ 4d] systems/mnemosyne.md +- [ 7d] systems/model-providers.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md + +## NO updated DATE (7) + +- SCHEMA.md +- experiments/active/example.md +- incidents/2026-07-29-mnemosyne-relay.md +- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- index.md +- log.md +- reports/uber-eats/2026-07.md + +TOTAL: 62 stale, 21 fresh, 7 no-date of 90 pages \ No newline at end of file diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index 1ac996b..c953db0 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -36,11 +36,11 @@ Primary agent memory store for Hermes. ## Database Stats (live, 2026-10-04) | Table | Rows | Notes | |---|---|---| -| `working_memory` | 752 | Active memories | -| `episodic_memory` | 1,039 | Consolidated summaries | -| `memory_embeddings` | 739 | Working-tier embeddings | +| `working_memory` | 773 | Active memories | +| `episodic_memory` | 1,047 | Consolidated summaries | +| `memory_embeddings` | 760 | Working-tier embeddings | | `memory` (legacy) | 2,181 | Old table, not migrated | -| `vec_working` / `vec_working_rowids` | 739 | Working vector index | +| `vec_working` / `vec_working_rowids` | 760 | Working vector index | | `vec_episodes` / `vec_episodes_rowids` | 1,035 | Episodic vector index | | `vec_facts_rowids` | 0 | `memoria_facts` is intentionally left unvectorised | | Shared surface | 0 | Empty — cross-agent sharing not functional | @@ -49,6 +49,8 @@ Primary agent memory store for Hermes. ## Known Issues +**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]]. + ### Vector embedding coverage gap — RESOLVED (2026-08-02) - Was: only 159/1,476 working memories (10.8%) embedded and 0 episodic, so recall fell back to FTS5 for ~90% of working and all episodic memories. - **Resolved** by the 2026-08-02 reindex pass. Live coverage on 2026-10-04 is **96.5% working / 99.4% episodic**, and new embeddings are still being generated. The figures above are kept as the before-state only. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 7af5506..9d2d29b 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-08 -verified_on: 2026-10-08 +updated: 2026-10-09 +verified_on: 2026-10-09 +review_after: 2026-10-23 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -118,3 +119,55 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - [[incidents/2026-09-28-hermes-managed-runtime-deps]] - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] + +--- + +## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09) + +**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at +2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day: +**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}. + +**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one +`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; +every run from 18:00 onward completed. Full timeline, job list and incident table: +[[incidents/2026-10-09-cron-provider-timeout-storm]]. + +- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05, + `ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`. +- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its + 03:00 run today decides whether the streak clears. +- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in + `agent.log`, so it reported the storm rather than suffering it. + +**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed +(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and +committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08 +00:12). The 2026-10-09 run covers both days. + +**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md` +modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit +step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator +ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output. + +**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26 +DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*. +Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`, +`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel +(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not +mutate cron state**; retiring the job is open item #1 below. + +**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; +capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, +1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**. +See [[incidents/2026-10-09-memory-recall-relevance]]. + +### Open items (2026-10-09) + +1. **Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed. +2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test. +3. **Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2). +4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit. +5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open. +6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open. + diff --git a/systems/skills-index.md b/systems/skills-index.md index 2b46dba..4710e63 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-07 -verified_on: 2026-10-07 +updated: 2026-10-08 +verified_on: 2026-10-08 confidence: high tags: [skills, hermes] sources: []