From 3271ff12a58c6994d863bfe71b2d76bf9ca3194e Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Sun, 25 Oct 2026 00:16:26 +0800 Subject: [PATCH] audit 2026-10-25: third consecutive clean cron day; new incident (agent refused 3 Conduit call requests); correct 10-22 memory-fix evidence; conduit 0.69.0/f000bf0 --- current-state.md | 35 ++++++- ...2026-10-22-conduit-memory-start-timeout.md | 22 ++++- ...2026-10-24-conduit-call-request-refused.md | 88 ++++++++++++++++++ index.md | 9 +- log.md | 45 +++++++++ systems/conduit.md | 92 +++++++++++++------ systems/scheduled-tasks.md | 51 +++++++++- 7 files changed, 296 insertions(+), 46 deletions(-) create mode 100644 incidents/2026-10-24-conduit-call-request-refused.md diff --git a/current-state.md b/current-state.md index 1750cc3..f379a22 100644 --- a/current-state.md +++ b/current-state.md @@ -3,18 +3,43 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-10-31 +updated: 2026-10-25 +verified_on: 2026-10-25 +review_after: 2026-11-01 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-24 +Updated: 2026-10-25 -## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) +## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) + +- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window + 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at + 170 / 0** (169 at the 10-24 audit). Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**, + **4 non-clean `last_status`**, **2 unclosed incident rows** — nothing opened, nothing closed + in-window — pins unchanged. **14 sessions: 11 `cron` + 3 `desktop`.** → + [[systems/scheduled-tasks]] +- **⚠ New: the agent refused to phone Anthony, three times.** On 2026-10-24 12:53–12:56 he asked to + be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`). All + three replies asserted the agent has **no calling capability** — with **zero tool calls** in the + session — while `conduit_call_user` was one `tool_search` away and his call settings were + permissive. **Capability asserted instead of discovered**; the tool itself is fine. → new page + [[incidents/2026-10-24-conduit-call-request-refused]] +- **Conduit, corrected and updated.** Plugin **`0.69.0` / HEAD `f000bf0` (22 commits on)** while the + gateway process still runs the 10-23 code and the memory-timeout patch is **uncommitted and now + five releases behind HEAD**. Device phone-status contact last 10-23 19:10:30; health ingestion to + 10-24 09:59:59; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** (the + 10-24 page's 308 / 62 was a narrower grouping). ⚠ **`conduit-calls.json → history` is empty** — the + 10-22 memory-fix incident's “two calls closed cleanly 00:03:23 / 00:06:57” was a **false positive** + (cron turn-ends, not calls), so that fix still has no positive call evidence. → + [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] +- **Also updated:** `systems/conduit.md`, `incidents/2026-10-22-conduit-memory-start-timeout.md` + (correction), `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.** + +## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) *(superseded by the 2026-10-25 block above)* - **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; diff --git a/incidents/2026-10-22-conduit-memory-start-timeout.md b/incidents/2026-10-22-conduit-memory-start-timeout.md index 3c92c92..ccecb67 100644 --- a/incidents/2026-10-22-conduit-memory-start-timeout.md +++ b/incidents/2026-10-22-conduit-memory-start-timeout.md @@ -3,9 +3,9 @@ title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout) type: incident status: resolved created: 2026-10-24 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-11-07 +updated: 2026-10-25 +verified_on: 2026-10-25 +review_after: 2026-11-08 confidence: high tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] @@ -76,14 +76,25 @@ touching the file. therefore running the patched code. - **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that restart (`grep` across `~/.hermes/logs/*.log`, deduped). -- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and - **00:06:57**, both `outcome: done`. +- ~~Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and + **00:06:57**, both `outcome: done`.~~ **Corrected 2026-10-25.** Those two + `conduit-calls.json → ends` entries are the **cron** sessions of that night (Mnemosyne Embedding + Coverage Monitor 00:01, Health Monitor 00:04), not calls; `ends` records *every* turn-end that + fires a watch (`MAX_RECENT_ENDS = 64`). **`history` — real call records — is empty**, so the fix + has **no positive call evidence** and no call has been placed since it landed. The surviving + evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it. - **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is not positive proof that recall was populated on those calls. The proven part is that the call is no longer abandoned by the memory layer. ## Follow-ups +- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json → + history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD** + (`0.64.0` → `0.69.0`). +- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which + asserted the capability did not exist **without calling the tool** → + [[incidents/2026-10-24-conduit-call-request-refused]]. - **The patch is uncommitted.** A plugin update that resets `~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream (`kaishi00/hermes-conduit-notifier`). @@ -92,6 +103,7 @@ touching the file. ## Related +- [[incidents/2026-10-24-conduit-call-request-refused]] - [[systems/conduit]] - [[systems/mnemosyne]] - [[incidents/2026-10-09-memory-recall-relevance]] diff --git a/incidents/2026-10-24-conduit-call-request-refused.md b/incidents/2026-10-24-conduit-call-request-refused.md new file mode 100644 index 0000000..30209c1 --- /dev/null +++ b/incidents/2026-10-24-conduit-call-request-refused.md @@ -0,0 +1,88 @@ +--- +title: Conduit Call Requests Refused by the Agent +type: incident +status: open +created: 2026-10-25 +updated: 2026-10-25 +verified_on: 2026-10-25 +review_after: 2026-11-08 +confidence: high +tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved] +sources: [live-sqlite-sessions, live-sqlite-conduit, plugin-tree, live-probe] +--- + +# Conduit Call Requests Refused by the Agent — 2026-10-24 + +**Window:** 2026-10-24 12:53 → 12:56 AWST (three attempts) +**Status:** Open — behavioural failure in the agent, **not** a broken tool or config +**Impact:** Anthony asked to be phoned three times to test Conduit calls. No call was placed, and +he was told the capability does not exist. + +## What happened + +Three `desktop` sessions inside three minutes, all asking for a call: + +| Session | Started | Model | Tool calls | Reply | +|---|---|---|---|---| +| `20261024_125329_08c71f` | 12:53:34 | `kilo-auto/free`, switched mid-session to `smart-route` (`custom:omni-101`) | **0** | *“I don't have the ability to make phone calls …”*, then *“There's no calling capability in my toolset”*, then *“I don't have a phone calling capability”* | +| `20261024_125538_d054bb` | 12:55:54 | `smart-route` | **0** | *“Hey, I'm here and listening. What can I do for you?”* — the greeting a call would open with, but no call was placed | + +Anthony's turns in `…08c71f`, verbatim: *“Call me now to test calls.”* → *“Yes. You do”* → *“CAL ME”*. +**No tool was invoked in either session** (`tool_call_count: 0` — no `tool_search`, no `skill_view`, +no `conduit_call_user`). Whether the app itself rang (tap-to-call) is not visible in the plugin's +records; what is proven is that the agent never attempted a call. + +## The capability does exist + +- **`conduit_call_user`** (toolset `conduit`, 📞) is registered by the plugin for **every** session + (`register_tool` in `call_tool.py`) and is reachable through Hermes's lazy tool discovery + (`tool_search`) — which is exactly how the nightly wiki-audit session can see it. +- The plugin ships the skill **`calling-the-user`** (`skill_view("conduit_push:calling-the-user")`), + written for this question. +- Anthony's call settings were **permissive at the time** (`conduit-calls.json → settings`): + `enabled`, `when_asked`, `decides`, `alerts` all `true`; `min_gap_s 120` / 6 per hour / 20 per day; + `avoid_unknown_place false`. A user-asked call would have gone out. +- The device was paired and the gateway (started 2026-10-23 18:54:43, carrying the memory-timeout + patch) was up throughout. + +So nothing outside the agent blocked the call. The tool was simply never called. + +## Root cause + +**Capability asserted instead of discovered.** Both models answered from priors (“I'm a text-based +AI agent”) rather than searching the tool registry or loading the bundled skill. The first reply came +from a free model, but the mid-session switch to `smart-route` did not change the answer — so this is +not one weak model's fault. The agent treated *“I don't know of such a tool”* as *“there is no such +tool”*, in a session where the tool was one `tool_search` away. + +## Evidence correction (affects the 10-22 memory-timeout incident) + +`~/.hermes/conduit-calls.json` is **not** a call log: + +- `history` — real call records — is **empty** (0 entries) at 2026-10-25 00:2x. +- `ends` (ring buffer, `MAX_RECENT_ENDS = 64` in `calls_store.py`) records **every turn-end that + fires a watch** (`fire()` appends `{session_id, outcome, at}`). At this audit its two entries are + **cron** sessions — `local-afac8b…` (Mnemosyne Embedding Coverage Monitor, 10-25 00:01) and + `local-8cfe41…` (Mnemosyne Health Monitor, 10-25 00:04). + +Therefore the 10-24 note that *“two call sessions closed cleanly at 00:03:23 / 00:06:57”* was a +**false positive**: those timestamps match the two Mnemosyne **cron** sessions that ran that night, +not calls. Do not read `ends` as “a call happened” — read `history`. The correction is also stamped +into [[incidents/2026-10-22-conduit-memory-start-timeout]]. + +## Open items + +- **Verify a real call end-to-end.** The memory-timeout fix has still never been exercised by a call + (`history` empty since it landed). Cheapest path: tap-to-call in the Conduit app; a chat-route call + needs the agent to actually invoke `conduit_call_user`. +- The failure is behavioural: worth checking whether the plugin's tool appears in a desktop session's + headline tool set at all (`sessions.tool_names` stores only a hash, so it could not be read from + `state.db` — the deferred-discovery path is the likely culprit for the miss). +- Consider surfacing `calling-the-user` more loudly (e.g. a system-prompt hint) so a model does not + have to discover it while a user is waiting. + +## Related + +- [[systems/conduit]] +- [[incidents/2026-10-22-conduit-memory-start-timeout]] +- [[concepts/tool-calling]] diff --git a/index.md b/index.md index 8978037..a54d431 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-24 +> model routing and automation. Last updated: 2026-10-25 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -32,14 +32,15 @@ - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations -- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, v`0.64.0` (HEAD `eaa14be`), upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device (iPhone app `2.0.0`). **Carries an uncommitted in-tree patch** — the memory-start timeout fix below. +- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.0` (HEAD `f000bf0` — 22 commits on since the 10-24 audit)**, upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`, process up since **2026-10-23 18:54:43** (so those upstream commits are **not** live); one registered device (iPhone app `2.0.0`; phone-status contact last 10-23 19:10, health ingestion to 10-24 09:59; store 569.9 MB / 1,520,229 events / 3,028 `buckets`). **Still carries the uncommitted in-tree patch — now five releases behind HEAD** (`conduit-calls.json → history` is **empty**, so the memory fix has no positive call evidence; the “two calls closed cleanly 00:03:23 / 00:06:57” reading was **wrong** — those were cron turn-ends). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool** → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **As of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents -- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it (two calls closed cleanly 00:03:23 / 00:06:57 on 10-24). **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] +- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]] +- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] diff --git a/log.md b/log.md index 33d68bc..d121866 100644 --- a/log.md +++ b/log.md @@ -1154,3 +1154,48 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than syncing it, since no script owns it. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. + +## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused + +- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled + at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs + (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows** + (`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) — + **nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3 + `desktop`.** +- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be + phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` → + `CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session. + `conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and + has a bundled skill (`calling-the-user`); the call settings were permissive + (`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted + instead of discovered** — not a missing tool, and not one weak model (the free model's answer was + repeated after a switch to `smart-route`). +- **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence + (“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**. + `conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch** + (`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne + Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is + **empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a + call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43 + restart. +- **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22 + commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the + uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the + `conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified + (`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59); + store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62 + `windows` was a **narrower grouping**, corrected rather than read as a collapse. +- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25` + section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked + superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22 + line). +- **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last + run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`, + `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are + the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system + obsolete, no decision reversed in the window. +- **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a + `+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59 + AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`. +- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only. diff --git a/systems/conduit.md b/systems/conduit.md index 966d349..94785bc 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,12 +3,12 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-11-07 +updated: 2026-10-25 +verified_on: 2026-10-25 +review_after: 2026-11-08 confidence: high -tags: [conduit, plugin, phone, iphone, push, voice, memory] -sources: [live-probe, plugin-tree, live-sqlite-conduit, live-log, session-20261023_182601] +tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling] +sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329] --- # Conduit @@ -23,12 +23,16 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste ## Where it lives - **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` -- **Version:** `0.64.0` (`manifest_version: 1`), git HEAD `eaa14be` — *“Pulse: mark follow-up cards - done, by hand or after Do it (0.64.0) (#323)”* +- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` — + *“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the + `0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse + cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway + restart, `channels.yaml`, App Store builds following the archived plugin version). - **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push) -- ⚠ **The working tree carries an uncommitted local patch** (`M dashboard/plugin_api.py`, the - memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]). A plugin update - that resets the tree drops it. +- ⚠ **The working tree still carries the uncommitted local patch** (`M dashboard/plugin_api.py`, the + memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]) — now **five releases + behind HEAD** (`0.64.0` → `0.69.0`), so the chance a plugin update resets the tree and drops it is + rising, not falling. - **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`, `reading-health-data`, `using-calendar`, `using-location`. - **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`, @@ -36,22 +40,46 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`, `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`. -## Current state (verified 2026-10-24 00:1x AWST) -- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** (this - start picked up the memory-timeout patch). -- **Device:** one registered — **iPhone, app version `2.0.0`**, features - `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards` - (last seen 2026-10-23 19:02:16). No other device is registered. +## Current state (verified 2026-10-25 00:2x–00:5x AWST) +- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and + **not restarted since** — so it is still the process that picked up the memory-timeout patch, and + the `0.64.0` → `0.69.0` upstream commits are **not** live. +- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`, + features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`; + tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**. + `last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at + this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were + recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow. + (The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.) - **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. -- **Recent calls:** two call sessions closed cleanly on **2026-10-24 00:03:23** and - **00:06:57** (both `outcome: done`, `conduit-calls.json` written 00:06:57) — the first - evidence after the memory-timeout fix. -- **Data volume:** `conduit-health.sqlite3` is **561.7 MB** (WAL 6.1 MB) holding - **1,501,165 `events`** rows across **308 `buckets`** / **62 `windows`** for its single device; - `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **No retention or - compaction has been observed** — the health store has grown from card/health ingestion alone - and is worth watching against the CT460 disk budget. +- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded + since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The + “two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false + positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records + any turn-end that fires a watch, not calls → + [[incidents/2026-10-24-conduit-call-request-refused]]. +- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h), + holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`), + **3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**, + newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. + ⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads + **3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or + compaction has been observed** — the store grows from ingestion alone and is worth watching against + the CT460 disk budget. + +## Calls from Hermes (the `conduit_call_user` tool) +The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and +ships the matching skill `calling-the-user`. Hermes exposes it through **lazy tool discovery** +(`tool_search`), so it is not in a session's headline tool list — it has to be looked up. +Gate conditions (`call_tool.py`): a real session (subagents are refused), a paired device, +`settings.enabled`, and either `when_asked` (with the ask recorded as user-originated) or `decides`; +the call rings when the asking turn ends and plays the reason first. Limits: `min_gap_s 120` / +6 per hour / 20 per day. + +⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which +asserted the capability did not exist **without calling the tool or loading the skill** → +[[incidents/2026-10-24-conduit-call-request-refused]]. ## Memory integration The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not @@ -74,17 +102,25 @@ ss -tlnp | grep 9119 # did a voice call lose its memory context? grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail +# did a real call happen? (history = calls; ends = turn-ends, NOT calls) +python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')" + # state files (no secrets in the wiki) ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ``` ## Open items -- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it. -- No retention policy on `conduit-health.sqlite3` (561.7 MB and growing). -- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* - was raised. → [[systems/mnemosyne]] +- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is + now five releases behind HEAD. +- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is + still unexercised by a real call, and the 10-24 refusal means the chat route has never worked → + [[incidents/2026-10-24-conduit-call-request-refused]]. +- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing). +- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was + raised. → [[systems/mnemosyne]] ## Related +- [[incidents/2026-10-24-conduit-call-request-refused]] - [[incidents/2026-10-22-conduit-memory-start-timeout]] - [[systems/mnemosyne]] - [[systems/scheduled-tasks]] diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index e60e915..398f210 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-24 -verified_on: 2026-10-24 -review_after: 2026-10-30 +updated: 2026-10-25 +verified_on: 2026-10-25 +review_after: 2026-10-31 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] +- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. +- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] - **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]] - **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]]. - **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] @@ -26,6 +27,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). +- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171. - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. @@ -532,3 +534,44 @@ working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[system **Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention. + +## Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x) + +**A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0.** Fleet +unchanged at **28 jobs (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident +rows**, pins unchanged. **Nothing opened and nothing closed in the incident ledger in-window.** 14 +sessions: 11 `cron` + 3 `desktop`. + +**The window's fault is off-cron.** On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to +test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); every reply asserted the +agent has **no calling capability**, with **zero tool calls** in the session — no `tool_search`, no +`skill_view`, no `conduit_call_user`. The tool exists, is registered for every session, is reachable +through lazy discovery, and the plugin ships the `calling-the-user` skill; his call settings +(`enabled` / `when_asked` / `decides` / `alerts`) were permissive and the device was paired. The first +reply came from `kilo-auto/free`, but the mid-session switch to `smart-route` repeated it — **capability +asserted instead of discovered**, not a broken tool → +[[incidents/2026-10-24-conduit-call-request-refused]]. + +**Conduit re-verified.** Plugin **`0.69.0` / HEAD `f000bf0`** (22 commits on from the `0.64.0` / +`eaa14be` recorded on 10-24) while the gateway process still dates from **2026-10-23 18:54:43**, so the +upstream commits are **not live** and the memory-timeout patch is **uncommitted and now five releases +behind HEAD**. Device phone-status contact last **10-23 19:10:30**; health ingestion ran to +**10-24 09:59:59**; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** — the 10-24 +page's 308 `buckets` / 62 `windows` was a **narrower grouping**, not a collapse. ⚠ **Evidence +correction:** `conduit-calls.json → history` is **empty**, and the 10-22 incident's “two calls closed +cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** turn-ends of that night (`ends` records +any turn-end that fires a watch, `MAX_RECENT_ENDS = 64`) — so the memory fix still has **no positive +call evidence** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. + +**Method note:** daily tallies are bucketed with `substr(started_at,1,10)` — `date(started_at)` +normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section). + +**Pages touched:** `incidents/2026-10-24-conduit-call-request-refused.md` (new), `systems/conduit.md`, +`incidents/2026-10-22-conduit-memory-start-timeout.md` (correction), `systems/scheduled-tasks.md`, +`current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no job retired, no system +obsolete, no decision reversed in the window. + +**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending +Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 +stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit +health store with no retention; and a **call that has never been placed** since the memory fix.