audit 2026-10-25: third consecutive clean cron day; new incident (agent refused 3 Conduit call requests); correct 10-22 memory-fix evidence; conduit 0.69.0/f000bf0

This commit is contained in:
Tony0410
2026-10-25 00:16:26 +08:00
parent ec69d107ae
commit 3271ff12a5
7 changed files with 296 additions and 46 deletions
+30 -5
View File
@@ -3,18 +3,43 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-10-31
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-01
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Current State
Updated: 2026-10-24
Updated: 2026-10-25
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST)
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST)
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
170 / 0** (169 at the 10-24 audit). Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**,
**4 non-clean `last_status`**, **2 unclosed incident rows** — nothing opened, nothing closed
in-window — pins unchanged. **14 sessions: 11 `cron` + 3 `desktop`.** →
[[systems/scheduled-tasks]]
- **⚠ New: the agent refused to phone Anthony, three times.** On 2026-10-24 12:53–12:56 he asked to
be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`). All
three replies asserted the agent has **no calling capability** — with **zero tool calls** in the
session — while `conduit_call_user` was one `tool_search` away and his call settings were
permissive. **Capability asserted instead of discovered**; the tool itself is fine. → new page
[[incidents/2026-10-24-conduit-call-request-refused]]
- **Conduit, corrected and updated.** Plugin **`0.69.0` / HEAD `f000bf0` (22 commits on)** while the
gateway process still runs the 10-23 code and the memory-timeout patch is **uncommitted and now
five releases behind HEAD**. Device phone-status contact last 10-23 19:10:30; health ingestion to
10-24 09:59:59; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** (the
10-24 page's 308 / 62 was a narrower grouping). ⚠ **`conduit-calls.json → history` is empty** — the
10-22 memory-fix incident's “two calls closed cleanly 00:03:23 / 00:06:57” was a **false positive**
(cron turn-ends, not calls), so that fix still has no positive call evidence. →
[[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- **Also updated:** `systems/conduit.md`, `incidents/2026-10-22-conduit-memory-start-timeout.md`
(correction), `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) *(superseded by the 2026-10-25 block above)*
- **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23:
**169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four;
@@ -3,9 +3,9 @@ title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident
status: resolved
created: 2026-10-24
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -76,14 +76,25 @@ touching the file.
therefore running the patched code.
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
**00:06:57**, both `outcome: done`.
- ~~Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
**00:06:57**, both `outcome: done`.~~ **Corrected 2026-10-25.** Those two
`conduit-calls.json → ends` entries are the **cron** sessions of that night (Mnemosyne Embedding
Coverage Monitor 00:01, Health Monitor 00:04), not calls; `ends` records *every* turn-end that
fires a watch (`MAX_RECENT_ENDS = 64`). **`history` — real call records — is empty**, so the fix
has **no positive call evidence** and no call has been placed since it landed. The surviving
evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer.
## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
(`0.64.0` → `0.69.0`).
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
(`kaishi00/hermes-conduit-notifier`).
@@ -92,6 +103,7 @@ touching the file.
## Related
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-10-09-memory-recall-relevance]]
@@ -0,0 +1,88 @@
---
title: Conduit Call Requests Refused by the Agent
type: incident
status: open
created: 2026-10-25
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
confidence: high
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
sources: [live-sqlite-sessions, live-sqlite-conduit, plugin-tree, live-probe]
---
# Conduit Call Requests Refused by the Agent — 2026-10-24
**Window:** 2026-10-24 12:53 → 12:56 AWST (three attempts)
**Status:** Open — behavioural failure in the agent, **not** a broken tool or config
**Impact:** Anthony asked to be phoned three times to test Conduit calls. No call was placed, and
he was told the capability does not exist.
## What happened
Three `desktop` sessions inside three minutes, all asking for a call:
| Session | Started | Model | Tool calls | Reply |
|---|---|---|---|---|
| `20261024_125329_08c71f` | 12:53:34 | `kilo-auto/free`, switched mid-session to `smart-route` (`custom:omni-101`) | **0** | *“I don't have the ability to make phone calls …”*, then *“There's no calling capability in my toolset”*, then *“I don't have a phone calling capability”* |
| `20261024_125538_d054bb` | 12:55:54 | `smart-route` | **0** | *“Hey, I'm here and listening. What can I do for you?”* — the greeting a call would open with, but no call was placed |
Anthony's turns in `…08c71f`, verbatim: *“Call me now to test calls.”* → *“Yes. You do”* → *“CAL ME”*.
**No tool was invoked in either session** (`tool_call_count: 0` — no `tool_search`, no `skill_view`,
no `conduit_call_user`). Whether the app itself rang (tap-to-call) is not visible in the plugin's
records; what is proven is that the agent never attempted a call.
## The capability does exist
- **`conduit_call_user`** (toolset `conduit`, 📞) is registered by the plugin for **every** session
(`register_tool` in `call_tool.py`) and is reachable through Hermes's lazy tool discovery
(`tool_search`) — which is exactly how the nightly wiki-audit session can see it.
- The plugin ships the skill **`calling-the-user`** (`skill_view("conduit_push:calling-the-user")`),
written for this question.
- Anthony's call settings were **permissive at the time** (`conduit-calls.json → settings`):
`enabled`, `when_asked`, `decides`, `alerts` all `true`; `min_gap_s 120` / 6 per hour / 20 per day;
`avoid_unknown_place false`. A user-asked call would have gone out.
- The device was paired and the gateway (started 2026-10-23 18:54:43, carrying the memory-timeout
patch) was up throughout.
So nothing outside the agent blocked the call. The tool was simply never called.
## Root cause
**Capability asserted instead of discovered.** Both models answered from priors (“I'm a text-based
AI agent”) rather than searching the tool registry or loading the bundled skill. The first reply came
from a free model, but the mid-session switch to `smart-route` did not change the answer — so this is
not one weak model's fault. The agent treated *“I don't know of such a tool”* as *“there is no such
tool”*, in a session where the tool was one `tool_search` away.
## Evidence correction (affects the 10-22 memory-timeout incident)
`~/.hermes/conduit-calls.json` is **not** a call log:
- `history` — real call records — is **empty** (0 entries) at 2026-10-25 00:2x.
- `ends` (ring buffer, `MAX_RECENT_ENDS = 64` in `calls_store.py`) records **every turn-end that
fires a watch** (`fire()` appends `{session_id, outcome, at}`). At this audit its two entries are
**cron** sessions — `local-afac8b…` (Mnemosyne Embedding Coverage Monitor, 10-25 00:01) and
`local-8cfe41…` (Mnemosyne Health Monitor, 10-25 00:04).
Therefore the 10-24 note that *“two call sessions closed cleanly at 00:03:23 / 00:06:57”* was a
**false positive**: those timestamps match the two Mnemosyne **cron** sessions that ran that night,
not calls. Do not read `ends` as “a call happened” — read `history`. The correction is also stamped
into [[incidents/2026-10-22-conduit-memory-start-timeout]].
## Open items
- **Verify a real call end-to-end.** The memory-timeout fix has still never been exercised by a call
(`history` empty since it landed). Cheapest path: tap-to-call in the Conduit app; a chat-route call
needs the agent to actually invoke `conduit_call_user`.
- The failure is behavioural: worth checking whether the plugin's tool appears in a desktop session's
headline tool set at all (`sessions.tool_names` stores only a hash, so it could not be read from
`state.db` — the deferred-discovery path is the likely culprit for the miss).
- Consider surfacing `calling-the-user` more loudly (e.g. a system-prompt hint) so a model does not
have to discover it while a user is waiting.
## Related
- [[systems/conduit]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[concepts/tool-calling]]
+5 -4
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-24
> model routing and automation. Last updated: 2026-10-25
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -32,14 +32,15 @@
- [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, v`0.64.0` (HEAD `eaa14be`), upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device (iPhone app `2.0.0`). **Carries an uncommitted in-tree patch** — the memory-start timeout fix below.
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.0` (HEAD `f000bf0` — 22 commits on since the 10-24 audit)**, upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`, process up since **2026-10-23 18:54:43** (so those upstream commits are **not** live); one registered device (iPhone app `2.0.0`; phone-status contact last 10-23 19:10, health ingestion to 10-24 09:59; store 569.9 MB / 1,520,229 events / 3,028 `buckets`). **Still carries the uncommitted in-tree patch — now five releases behind HEAD** (`conduit-calls.json → history` is **empty**, so the memory fix has no positive call evidence; the “two calls closed cleanly 00:03:23 / 00:06:57” reading was **wrong** — those were cron turn-ends). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool** → [[incidents/2026-10-24-conduit-call-request-refused]].
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/scheduled-tasks]] — **As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/scheduled-tasks]] — **As of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it (two calls closed cleanly 00:03:23 / 00:06:57 on 10-24). **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
+45
View File
@@ -1154,3 +1154,48 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
syncing it, since no script owns it.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled
at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs
(25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows**
(`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) —
**nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3
`desktop`.**
- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be
phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` →
`CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session.
`conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and
has a bundled skill (`calling-the-user`); the call settings were permissive
(`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted
instead of discovered** — not a missing tool, and not one weak model (the free model's answer was
repeated after a switch to `smart-route`).
- **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence
(“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**.
`conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch**
(`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne
Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is
**empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a
call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43
restart.
- **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22
commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the
uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the
`conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified
(`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59);
store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62
`windows` was a **narrower grouping**, corrected rather than read as a collapse.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25`
section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked
superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22
line).
- **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`,
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are
the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a
`+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59
AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only.
+64 -28
View File
@@ -3,12 +3,12 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-11-07
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-log, session-20261023_182601]
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329]
---
# Conduit
@@ -23,12 +23,16 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
- **Version:** `0.64.0` (`manifest_version: 1`), git HEAD `eaa14be` — *“Pulse: mark follow-up cards
done, by hand or after Do it (0.64.0) (#323)”*
- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
restart, `channels.yaml`, App Store builds following the archived plugin version).
- **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push)
- ⚠ **The working tree carries an uncommitted local patch** (`M dashboard/plugin_api.py`, the
memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]). A plugin update
that resets the tree drops it.
- ⚠ **The working tree still carries the uncommitted local patch** (`M dashboard/plugin_api.py`, the
memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]) — now **five releases
behind HEAD** (`0.64.0` → `0.69.0`), so the chance a plugin update resets the tree and drops it is
rising, not falling.
- **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`,
`reading-health-data`, `using-calendar`, `using-location`.
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
@@ -36,22 +40,46 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
## Current state (verified 2026-10-24 00:1x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** (this
start picked up the memory-timeout patch).
- **Device:** one registered — **iPhone, app version `2.0.0`**, features
`chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`
(last seen 2026-10-23 19:02:16). No other device is registered.
## Current state (verified 2026-10-25 00:2x–00:5x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and
**not restarted since** — so it is still the process that picked up the memory-timeout patch, and
the `0.64.0` → `0.69.0` upstream commits are **not** live.
- **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**.
`last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at
this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were
recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow.
(The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.)
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
- **Recent calls:** two call sessions closed cleanly on **2026-10-24 00:03:23** and
**00:06:57** (both `outcome: done`, `conduit-calls.json` written 00:06:57) — the first
evidence after the memory-timeout fix.
- **Data volume:** `conduit-health.sqlite3` is **561.7 MB** (WAL 6.1 MB) holding
**1,501,165 `events`** rows across **308 `buckets`** / **62 `windows`** for its single device;
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **No retention or
compaction has been observed** — the health store has grown from card/health ingestion alone
and is worth watching against the CT460 disk budget.
- **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded
since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The
“two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false
positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records
any turn-end that fires a watch, not calls →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h),
holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`),
**3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**,
newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB.
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
**3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or
compaction has been observed** — the store grows from ingestion alone and is worth watching against
the CT460 disk budget.
## Calls from Hermes (the `conduit_call_user` tool)
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
ships the matching skill `calling-the-user`. Hermes exposes it through **lazy tool discovery**
(`tool_search`), so it is not in a session's headline tool list — it has to be looked up.
Gate conditions (`call_tool.py`): a real session (subagents are refused), a paired device,
`settings.enabled`, and either `when_asked` (with the ask recorded as user-originated) or `decides`;
the call rings when the asking turn ends and plays the reason first. Limits: `min_gap_s 120` /
6 per hour / 20 per day.
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool or loading the skill** →
[[incidents/2026-10-24-conduit-call-request-refused]].
## Memory integration
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
@@ -74,17 +102,25 @@ ss -tlnp | grep 9119
# did a voice call lose its memory context?
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
# state files (no secrets in the wiki)
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
```
## Open items
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it.
- No retention policy on `conduit-health.sqlite3` (561.7 MB and growing).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget*
was raised. → [[systems/mnemosyne]]
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
now five releases behind HEAD.
- **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is
still unexercised by a real call, and the 10-24 refusal means the chat route has never worked →
[[incidents/2026-10-24-conduit-call-request-refused]].
- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
raised. → [[systems/mnemosyne]]
## Related
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[systems/mnemosyne]]
- [[systems/scheduled-tasks]]
+47 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-24
verified_on: 2026-10-24
review_after: 2026-10-30
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-10-31
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
@@ -26,6 +27,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
@@ -532,3 +534,44 @@ working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[system
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped;
the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
## Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)
**A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0.** Fleet
unchanged at **28 jobs (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident
rows**, pins unchanged. **Nothing opened and nothing closed in the incident ledger in-window.** 14
sessions: 11 `cron` + 3 `desktop`.
**The window's fault is off-cron.** On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to
test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); every reply asserted the
agent has **no calling capability**, with **zero tool calls** in the session — no `tool_search`, no
`skill_view`, no `conduit_call_user`. The tool exists, is registered for every session, is reachable
through lazy discovery, and the plugin ships the `calling-the-user` skill; his call settings
(`enabled` / `when_asked` / `decides` / `alerts`) were permissive and the device was paired. The first
reply came from `kilo-auto/free`, but the mid-session switch to `smart-route` repeated it — **capability
asserted instead of discovered**, not a broken tool →
[[incidents/2026-10-24-conduit-call-request-refused]].
**Conduit re-verified.** Plugin **`0.69.0` / HEAD `f000bf0`** (22 commits on from the `0.64.0` /
`eaa14be` recorded on 10-24) while the gateway process still dates from **2026-10-23 18:54:43**, so the
upstream commits are **not live** and the memory-timeout patch is **uncommitted and now five releases
behind HEAD**. Device phone-status contact last **10-23 19:10:30**; health ingestion ran to
**10-24 09:59:59**; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** — the 10-24
page's 308 `buckets` / 62 `windows` was a **narrower grouping**, not a collapse. ⚠ **Evidence
correction:** `conduit-calls.json → history` is **empty**, and the 10-22 incident's “two calls closed
cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** turn-ends of that night (`ends` records
any turn-end that fires a watch, `MAX_RECENT_ENDS = 64`) — so the memory fix still has **no positive
call evidence** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
**Method note:** daily tallies are bucketed with `substr(started_at,1,10)` — `date(started_at)`
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).
**Pages touched:** `incidents/2026-10-24-conduit-call-request-refused.md` (new), `systems/conduit.md`,
`incidents/2026-10-22-conduit-memory-start-timeout.md` (correction), `systems/scheduled-tasks.md`,
`current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no job retired, no system
obsolete, no decision reversed in the window.
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
health store with no retention; and a **call that has never been placed** since the memory fix.