From 573a4a2434d6a43df550fbf713e71c42c105a1d5 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:24:44 +0800 Subject: [PATCH] audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages --- current-state.md | 43 +++++++- incidents/2026-07-29-mnemosyne-relay.md | 13 +++ ...04-omniroute-free-deepseek-silent-stops.md | 13 +++ .../2026-10-09-cron-provider-timeout-storm.md | 17 ++-- ...2026-10-22-conduit-memory-start-timeout.md | 97 +++++++++++++++++++ index.md | 12 ++- log.md | 47 +++++++++ systems/conduit.md | 91 +++++++++++++++++ systems/mnemosyne.md | 22 ++++- systems/scheduled-tasks.md | 52 +++++++++- 10 files changed, 384 insertions(+), 23 deletions(-) create mode 100644 incidents/2026-10-22-conduit-memory-start-timeout.md create mode 100644 systems/conduit.md diff --git a/current-state.md b/current-state.md index 479f0d9..1750cc3 100644 --- a/current-state.md +++ b/current-state.md @@ -3,18 +3,51 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-23 -verified_on: 2026-10-23 -review_after: 2026-10-30 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-10-31 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-23 +Updated: 2026-10-24 -## Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST) +## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) + +- **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: + **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; + **2026-10-22 settled at 173 / 0**. `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00) + ran clean on its 10-23 slot, closing the storm's **last two incident rows** + (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`) at **08:04** — all 12 rows closed, job `ok` again. + **Unclosed incident rows 4 → 2**, **non-clean `last_status` 5 → 4**, **fleet unchanged at 28 jobs + (25 enabled / 3 disabled)**, pins unchanged. → [[incidents/2026-10-09-cron-provider-timeout-storm]], + [[systems/scheduled-tasks]] +- **New: Conduit is documented.** The dashboard plugin + iOS app that puts Hermes on Anthony's phone + (voice calls, cards, push) now has a system page — plugin tree, version `0.64.0` (HEAD `eaa14be`), + upstream `kaishi00/hermes-conduit-notifier`, gateway on `:9119`, one registered device (iPhone, app + `2.0.0`), its state/DB files and their sizes, and the uncommitted patch it carries. + → [[systems/conduit]] +- **⚠ New fault, found and fixed in-window: Conduit voice calls were being abandoned by the memory + layer.** The plugin hard-coded a **10 s** budget for one request *including the Mnemosyne provider + start*, while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls hit it + (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed + out; abandoning the call`) and ran **without recall**. Patched in the plugin's working tree to a **60 s** + default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that proceeds without recall + instead of aborting; the dashboard restarted **18:54:43** and **no abort line exists after it**, with two + calls closing cleanly at 00:03:23 and 00:06:57 on 10-24. **The patch is uncommitted** — a plugin update + reverts it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/mnemosyne]] +- **Sessions: 14 in the window** — 13 `cron` + 1 `desktop` (10-23 18:26:12, the phone-call session above); + no other interactive sessions. +- **Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed + (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] +- **Housekeeping:** frontmatter added to `incidents/2026-07-29-mnemosyne-relay.md` and + `incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md`, both of which sat outside the SCHEMA + (rule 4). Freshness: 102 pages; **3** carry a `verified_on` older than 21 days — `infrastructure/hosts.md`, + `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` (all 94 d). Nothing removed or archived. + +## Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST) *(superseded by the 2026-10-24 entry above)* - **First zero-failure cron day in three days, and five incidents cleared at once.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`); 4 runs had already completed on 10-23 by 00:0x. **Five diff --git a/incidents/2026-07-29-mnemosyne-relay.md b/incidents/2026-07-29-mnemosyne-relay.md index b2a393b..e072454 100644 --- a/incidents/2026-07-29-mnemosyne-relay.md +++ b/incidents/2026-07-29-mnemosyne-relay.md @@ -1,3 +1,16 @@ +--- +title: Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29 +type: incident +status: resolved +created: 2026-07-29 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 +confidence: high +tags: [mnemosyne, config, relay, incident, resolved] +sources: [live-probe, config, session-20260729] +--- + # mnemosyne-tools-none — 2026-07-29 ## FIXED: `memory.mnemosyne.tools: None` in config.yaml diff --git a/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md b/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md index 5fea4dc..e278782 100644 --- a/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +++ b/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md @@ -1,3 +1,16 @@ +--- +title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04 +type: incident +status: monitoring +created: 2026-08-04 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 +confidence: high +tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated] +sources: [live-probe, omniroute-log, session-20260804] +--- + # [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool) ## Status diff --git a/incidents/2026-10-09-cron-provider-timeout-storm.md b/incidents/2026-10-09-cron-provider-timeout-storm.md index 433c417..af484d0 100644 --- a/incidents/2026-10-09-cron-provider-timeout-storm.md +++ b/incidents/2026-10-09-cron-provider-timeout-storm.md @@ -1,22 +1,23 @@ --- title: Fleet-Wide Provider Timeout Storm — 2026-10-09 type: incident -status: unresolved +status: resolved created: 2026-10-09 -updated: 2026-10-13 -verified_on: 2026-10-13 -review_after: 2026-10-16 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 confidence: high -tags: [cron, omniroute, router, timeout, incident, unresolved] +tags: [cron, omniroute, router, timeout, incident, resolved] sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1] --- # Fleet-Wide Provider Timeout Storm — 2026-10-09 **Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed) -**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only -`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot -(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic. +**Status:** Resolved — **all 12 of its incident rows are closed.** The last hold-outs +(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job) +closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again. +The storm contributes **no** unclosed rows. Nothing here is still symptomatic. **Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08** ## What happened diff --git a/incidents/2026-10-22-conduit-memory-start-timeout.md b/incidents/2026-10-22-conduit-memory-start-timeout.md new file mode 100644 index 0000000..3c92c92 --- /dev/null +++ b/incidents/2026-10-22-conduit-memory-start-timeout.md @@ -0,0 +1,97 @@ +--- +title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) +type: incident +status: resolved +created: 2026-10-24 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 +confidence: high +tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] +sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] +--- + +# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23 + +**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences) +**Status:** Resolved 2026-10-23 18:54 (patched and verified) +**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its +memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's +memory layer. + +## Symptom + +Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt +that hit a cold memory provider: + +``` +2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call +2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call +2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call +2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out +``` + +## Root cause + +`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a +budget for **one request including the provider start**. Mnemosyne's cold start for the +`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the +budget. The plugin then ran the call **without memory context**, and the failure was logged as +`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no +error surfaced to the caller. + +This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and +the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]] + +## Fix applied (2026-10-23, live before 18:54:43) + +A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still +uncommitted in the working tree**: + +```diff +-MEMORY_TIMEOUT_S = 10.0 ++# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the ++# conduit-voice session, so give it enough room; tunable via env. ++MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0")) +``` + +and a hard-failure guard so a token/start error degrades instead of aborting: + +```diff +- entry = _memory_providers.get(key, name, deadline) ++ try: ++ entry = _memory_providers.get(key, name, deadline) ++ except TokenError: ++ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name) ++ entry = None +``` + +The new default is used by every deadline in the memory layer (6 call sites) plus the +`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without +touching the file. + +## Verification + +- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and + therefore running the patched code. +- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that + restart (`grep` across `~/.hermes/logs/*.log`, deduped). +- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and + **00:06:57**, both `outcome: done`. +- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is + not positive proof that recall was populated on those calls. The proven part is that the call + is no longer abandoned by the memory layer. + +## Follow-ups + +- **The patch is uncommitted.** A plugin update that resets + `~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream + (`kaishi00/hermes-conduit-notifier`). +- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised. + Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]] + +## Related + +- [[systems/conduit]] +- [[systems/mnemosyne]] +- [[incidents/2026-10-09-memory-recall-relevance]] diff --git a/index.md b/index.md index ca53b9d..8978037 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-22 +> model routing and automation. Last updated: 2026-10-24 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -32,20 +32,22 @@ - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations +- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, v`0.64.0` (HEAD `eaa14be`), upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device (iPhone app `2.0.0`). **Carries an uncommitted in-tree patch** — the memory-start timeout fix below. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** -- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). +- [[systems/scheduled-tasks]] — **As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` **`alerted`, unclosed**; the only cron failure of the day (171 `completed` / 1 `failed`). **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] +- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it (two calls closed cleanly 00:03:23 / 00:06:57 on 10-24). **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] +- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**. - [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) -- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) - [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05) - [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic diff --git a/log.md b/log.md index 13131c1..33d68bc 100644 --- a/log.md +++ b/log.md @@ -1107,3 +1107,50 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st `systems/model-providers.md` (22 d, updated 2026-10-01). The latter needs a review pass against the live OmniRoute catalogue before its next `verified_on`. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. + +## [2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST + +- **New page:** `systems/conduit.md` — the Conduit dashboard plugin + iOS app (voice calls, cards, + push). Plugin tree `~/.hermes/plugins/conduit_push`, version `0.64.0` (HEAD `eaa14be`, no tags + upstream), remote `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device + (iPhone, app `2.0.0`); state/DB files and sizes recorded. Where the wiki previously had no page for + the phone surface at all. +- **New incident:** [[incidents/2026-10-22-conduit-memory-start-timeout]] — three + `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call` warnings + (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s `MEMORY_TIMEOUT_S` in the + plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree + to a 60 s default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades + instead of aborting; the dashboard restarted 18:54:43 and **no abort line exists after it** (two + calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). **The patch is uncommitted.** +- **`incidents/2026-10-09-cron-provider-timeout-storm.md` → `status: resolved`.** Its last two rows + (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on `1b23c042e17d`) closed **2026-10-23 08:04** + when that weekly job's Friday slot ran clean; the job reads `ok` again. **All 12 rows closed** — + this incident no longer contributes an unclosed row anywhere. +- **Cron ledger:** window **2026-10-23 closed 169 `completed` / 0 `failed`** (+ this audit `running`) — + the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. Fleet **28 jobs (25 enabled / + 3 disabled)**, unchanged for a sixth audit. **Unclosed incident rows 4 → 2** (`c0a9fa_764558b5ae83` + `alerted` since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and + `0e753e_e7f8194a7d06` `detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4** + (`c0a9fa314542`, `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Pins unchanged (11 `omni-101` / + `smart-route`, 8 unpinned, 6 `omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 + `deepseek-paid-smart` = this audit). **14 sessions:** 13 `cron` + 1 `desktop` (18:26:12, the Conduit + phone-call session that exposed the fault above). +- **`systems/mnemosyne.md`:** new block recording the 10–45 s cold start and the consumer-timeout rule + (a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold + start itself was **not** optimised — only the budget was raised. +- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-24` + section), `current-state.md` (new Recent Changes block; 10-23 block marked superseded), `index.md` + (date, new `systems/conduit` and incident lines, refresh of the scheduled-tasks / skills-index / + storm blurbs), and **frontmatter added** to `incidents/2026-07-29-mnemosyne-relay.md` and + `incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md` — both sat outside the SCHEMA + (rule 4: every page has frontmatter). +- **Freshness:** 102 pages scanned at the last stocktake; **3 carry a `verified_on` older than 21 days** + — `infrastructure/hosts.md`, `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` + (all 94 d, unchanged). **Nothing removed or archived** this cycle: no job retired, no system + obsolete, no decision reversed in the window. +- **Note (hygiene):** a second, older copy of this wiki sits at `/home/hermes/wiki/agent-estate-wiki` + (101 pages, 3 incidents, no git remote, last commit `c798304` "mirror(2026-10-22 maintenance)"). + It is **not** served — `hermes-wiki-serve.service` serves `public/` from *this* repo — so it is a + stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than + syncing it, since no script owns it. +- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. diff --git a/systems/conduit.md b/systems/conduit.md new file mode 100644 index 0000000..966d349 --- /dev/null +++ b/systems/conduit.md @@ -0,0 +1,91 @@ +--- +title: Conduit +type: system +status: active +created: 2026-10-24 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 +confidence: high +tags: [conduit, plugin, phone, iphone, push, voice, memory] +sources: [live-probe, plugin-tree, live-sqlite-conduit, live-log, session-20261023_182601] +--- + +# Conduit + +## Purpose +Hermes on Anthony's phone. A dashboard plugin + iOS app pair that pushes cards, prompts, +location and health data between the Hermes gateway and the iPhone, and lets a **voice call** +be started from the phone where the agent can speak and listen (`conduit-voice` sessions). + +Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]] +(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge). + +## Where it lives +- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` +- **Version:** `0.64.0` (`manifest_version: 1`), git HEAD `eaa14be` — *“Pulse: mark follow-up cards + done, by hand or after Do it (0.64.0) (#323)”* +- **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push) +- ⚠ **The working tree carries an uncommitted local patch** (`M dashboard/plugin_api.py`, the + memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]). A plugin update + that resets the tree drops it. +- **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`, + `reading-health-data`, `using-calendar`, `using-location`. +- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`, + `conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id / + installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`, + `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`. + +## Current state (verified 2026-10-24 00:1x AWST) +- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** (this + start picked up the memory-timeout patch). +- **Device:** one registered — **iPhone, app version `2.0.0`**, features + `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards` + (last seen 2026-10-23 19:02:16). No other device is registered. +- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits + `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. +- **Recent calls:** two call sessions closed cleanly on **2026-10-24 00:03:23** and + **00:06:57** (both `outcome: done`, `conduit-calls.json` written 00:06:57) — the first + evidence after the memory-timeout fix. +- **Data volume:** `conduit-health.sqlite3` is **561.7 MB** (WAL 6.1 MB) holding + **1,501,165 `events`** rows across **308 `buckets`** / **62 `windows`** for its single device; + `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **No retention or + compaction has been observed** — the health store has grown from card/health ingestion alone + and is worth watching against the CT460 disk budget. + +## Memory integration +The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not +through Mnemosyne's `system_prompt_block()` — recall is wrapped in a per-request deadline +(`MEMORY_TIMEOUT_S`, default **60 s** as of 2026-10-23, overridable with +`CONDUIT_MEMORY_TIMEOUT_S`; it was a hard-coded **10 s** before that, which was too short for +Mnemosyne's 10–45 s cold start). If the provider fails to start, the plugin logs a warning and +proceeds without recall rather than abandoning the call. +See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyne]]. + +## Maintenance commands +```bash +# what version is installed / is the tree patched? +git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \ +git -C ~/.hermes/plugins/conduit_push status --short + +# is the gateway up, and since when? +ss -tlnp | grep 9119 + +# did a voice call lose its memory context? +grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail + +# state files (no secrets in the wiki) +ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 +``` + +## Open items +- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it. +- No retention policy on `conduit-health.sqlite3` (561.7 MB and growing). +- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* + was raised. → [[systems/mnemosyne]] + +## Related +- [[incidents/2026-10-22-conduit-memory-start-timeout]] +- [[systems/mnemosyne]] +- [[systems/scheduled-tasks]] +- [[current-state]] diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index d6f45af..d0c3b6a 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,9 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-19 -verified_on: 2026-10-19 +updated: 2026-10-24 +verified_on: 2026-10-24 +review_after: 2026-11-07 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -157,3 +158,20 @@ Primary agent memory store for Hermes. 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e. one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on every slot since (latest 10-18 00:30). + +## Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST) + +- **Cold start is the new failure vector for consumers: 10–45 s.** The Conduit voice plugin + (`conduit_push`) gave one request — provider start included — a hard-coded **10 s** budget + (`MEMORY_TIMEOUT_S`). Mnemosyne's cold start for the `conduit-voice` session regularly exceeds + that, so **three calls were abandoned by the memory layer** (10-22 11:28:17, 10-22 13:16:37, + 10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`), + and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default + **60 s**, overridable with `CONDUIT_MEMORY_TIMEOUT_S`, plus a `TokenError` guard that logs and + proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists + after it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]] +- **Rule for other consumers:** a 10 s memory budget is below the observed cold-start floor. Any + new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall" + rather than fail. The **underlying cold start was not optimised** — only the budget was raised. +- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic + 1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 5ff45eb..e60e915 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-23 -verified_on: 2026-10-23 +updated: 2026-10-24 +verified_on: 2026-10-24 review_after: 2026-10-30 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-23 00:1x AWST (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]] +- **Current as of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] +- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]] - **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]]. - **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] @@ -486,3 +487,48 @@ OmniRoute node — the durable fix is a second provider pin or a router-side fai call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows; Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard. + +## Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x) + +**Count.** As of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`): **28 jobs — +25 enabled / 3 disabled**, unchanged for a sixth consecutive audit. Window **2026-10-23: 169 `completed` / +0 `failed`** (+ this audit `running`), the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. + +**The 2026-10-09 timeout storm closed.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00) +ran clean on its 10-23 slot, closing the storm's last two rows (`1b23c0_c35703f637b8`, +`1b23c0_103cbe04a79b`) at **08:04**. The storm now holds **zero** unclosed rows and the job reads `ok` +again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. + +**Incident ledger: unclosed 4 → 2** (no rows opened, two closed). Remaining: `c0a9fa_764558b5ae83` +(`alerted` since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and +`0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01 — Facebook Birthday monthly scrape). + +**Non-clean `last_status`: 5 → 4** — `c0a9fa314542` (`error`, streak 1, Mon 10-26), `0e753e9720fd` +(`error`, 0, 11-01), `fad5a86979d4` (`delivery_failed`, Mon 10-26), `066bb5191280` (`delivery_failed`, 11-01). +No new failure signature appeared in the window. + +**Pins unchanged** — 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 +`custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / +`deepseek-paid-smart` (this audit). + +**Sessions: 14 in the window** — 13 `cron` + **1 `desktop`** (2026-10-23 18:26:12, 13 messages, the Conduit +phone-call session that exposed the memory-timeout fault below). + +**New fault found and fixed in-window (not a cron job):** the **Conduit** voice plugin gave the Mnemosyne +provider start a hard-coded **10 s** budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) +were abandoned by the memory layer and ran without recall. Patched the same evening to a **60 s** default +(`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades instead of aborting; dashboard +restarted 18:54:43 and **no abort line exists after it**. The patch is **uncommitted** in the plugin's +working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. + +**Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed +(`250c735`: freshness 2 stale / 94 fresh, skills-index 306). + +**Pages touched:** `systems/conduit.md` (new), `incidents/2026-10-22-conduit-memory-start-timeout.md` (new), +`incidents/2026-10-09-cron-provider-timeout-storm.md` (status → resolved), `systems/mnemosyne.md`, +`current-state.md`, `index.md`, `log.md`, plus frontmatter added to two legacy incident pages (see log). +**Nothing removed or archived:** no job retired, no system obsolete in the window. + +**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending +Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; +the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.