audit 2026-10-25: third consecutive clean cron day; new incident (agent refused 3 Conduit call requests); correct 10-22 memory-fix evidence; conduit 0.69.0/f000bf0

This commit is contained in:
Tony0410
2026-10-25 00:16:26 +08:00
parent ec69d107ae
commit 3271ff12a5
7 changed files with 296 additions and 46 deletions
+30 -5
View File
@@ -3,18 +3,43 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-24 updated: 2026-10-25
verified_on: 2026-10-24 verified_on: 2026-10-25
review_after: 2026-10-31 review_after: 2026-11-01
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
--- ---
# Current State # Current State
Updated: 2026-10-24 Updated: 2026-10-25
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) ## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST)
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
170 / 0** (169 at the 10-24 audit). Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**,
**4 non-clean `last_status`**, **2 unclosed incident rows** — nothing opened, nothing closed
in-window — pins unchanged. **14 sessions: 11 `cron` + 3 `desktop`.** →
[[systems/scheduled-tasks]]
- **⚠ New: the agent refused to phone Anthony, three times.** On 2026-10-24 12:53–12:56 he asked to
be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`). All
three replies asserted the agent has **no calling capability** — with **zero tool calls** in the
session — while `conduit_call_user` was one `tool_search` away and his call settings were
permissive. **Capability asserted instead of discovered**; the tool itself is fine. → new page
[[incidents/2026-10-24-conduit-call-request-refused]]
- **Conduit, corrected and updated.** Plugin **`0.69.0` / HEAD `f000bf0` (22 commits on)** while the
gateway process still runs the 10-23 code and the memory-timeout patch is **uncommitted and now
five releases behind HEAD**. Device phone-status contact last 10-23 19:10:30; health ingestion to
10-24 09:59:59; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** (the
10-24 page's 308 / 62 was a narrower grouping). ⚠ **`conduit-calls.json → history` is empty** — the
10-22 memory-fix incident's “two calls closed cleanly 00:03:23 / 00:06:57” was a **false positive**
(cron turn-ends, not calls), so that fix still has no positive call evidence. →
[[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- **Also updated:** `systems/conduit.md`, `incidents/2026-10-22-conduit-memory-start-timeout.md`
(correction), `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) *(superseded by the 2026-10-25 block above)*
- **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: - **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23:
**169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four;
@@ -3,9 +3,9 @@ title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
type: incident type: incident
status: resolved status: resolved
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-24 updated: 2026-10-25
verified_on: 2026-10-24 verified_on: 2026-10-25
review_after: 2026-11-07 review_after: 2026-11-08
confidence: high confidence: high
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved] tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff] sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
@@ -76,14 +76,25 @@ touching the file.
therefore running the patched code. therefore running the patched code.
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that - **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
restart (`grep` across `~/.hermes/logs/*.log`, deduped). restart (`grep` across `~/.hermes/logs/*.log`, deduped).
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and - ~~Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
**00:06:57**, both `outcome: done`. **00:06:57**, both `outcome: done`.~~ **Corrected 2026-10-25.** Those two
`conduit-calls.json → ends` entries are the **cron** sessions of that night (Mnemosyne Embedding
Coverage Monitor 00:01, Health Monitor 00:04), not calls; `ends` records *every* turn-end that
fires a watch (`MAX_RECENT_ENDS = 64`). **`history` — real call records — is empty**, so the fix
has **no positive call evidence** and no call has been placed since it landed. The surviving
evidence is the absence of abort lines after the 18:54:43 restart, and nothing has falsified it.
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is - **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
not positive proof that recall was populated on those calls. The proven part is that the call not positive proof that recall was populated on those calls. The proven part is that the call
is no longer abandoned by the memory layer. is no longer abandoned by the memory layer.
## Follow-ups ## Follow-ups
- ⚠ **The fix is still unexercised by a call** (see the correction above): `conduit-calls.json →
history` is empty at 2026-10-25 00:2x, and the plugin is now **five releases behind HEAD**
(`0.64.0` → `0.69.0`).
- **Follow-on failure (2026-10-24):** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool** →
[[incidents/2026-10-24-conduit-call-request-refused]].
- **The patch is uncommitted.** A plugin update that resets - **The patch is uncommitted.** A plugin update that resets
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream `~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
(`kaishi00/hermes-conduit-notifier`). (`kaishi00/hermes-conduit-notifier`).
@@ -92,6 +103,7 @@ touching the file.
## Related ## Related
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[systems/conduit]] - [[systems/conduit]]
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[incidents/2026-10-09-memory-recall-relevance]] - [[incidents/2026-10-09-memory-recall-relevance]]
@@ -0,0 +1,88 @@
---
title: Conduit Call Requests Refused by the Agent
type: incident
status: open
created: 2026-10-25
updated: 2026-10-25
verified_on: 2026-10-25
review_after: 2026-11-08
confidence: high
tags: [conduit, voice, tool-calling, agent-conduct, incident, unresolved]
sources: [live-sqlite-sessions, live-sqlite-conduit, plugin-tree, live-probe]
---
# Conduit Call Requests Refused by the Agent — 2026-10-24
**Window:** 2026-10-24 12:53 → 12:56 AWST (three attempts)
**Status:** Open — behavioural failure in the agent, **not** a broken tool or config
**Impact:** Anthony asked to be phoned three times to test Conduit calls. No call was placed, and
he was told the capability does not exist.
## What happened
Three `desktop` sessions inside three minutes, all asking for a call:
| Session | Started | Model | Tool calls | Reply |
|---|---|---|---|---|
| `20261024_125329_08c71f` | 12:53:34 | `kilo-auto/free`, switched mid-session to `smart-route` (`custom:omni-101`) | **0** | *“I don't have the ability to make phone calls …”*, then *“There's no calling capability in my toolset”*, then *“I don't have a phone calling capability”* |
| `20261024_125538_d054bb` | 12:55:54 | `smart-route` | **0** | *“Hey, I'm here and listening. What can I do for you?”* — the greeting a call would open with, but no call was placed |
Anthony's turns in `…08c71f`, verbatim: *“Call me now to test calls.”* → *“Yes. You do”* → *“CAL ME”*.
**No tool was invoked in either session** (`tool_call_count: 0` — no `tool_search`, no `skill_view`,
no `conduit_call_user`). Whether the app itself rang (tap-to-call) is not visible in the plugin's
records; what is proven is that the agent never attempted a call.
## The capability does exist
- **`conduit_call_user`** (toolset `conduit`, 📞) is registered by the plugin for **every** session
(`register_tool` in `call_tool.py`) and is reachable through Hermes's lazy tool discovery
(`tool_search`) — which is exactly how the nightly wiki-audit session can see it.
- The plugin ships the skill **`calling-the-user`** (`skill_view("conduit_push:calling-the-user")`),
written for this question.
- Anthony's call settings were **permissive at the time** (`conduit-calls.json → settings`):
`enabled`, `when_asked`, `decides`, `alerts` all `true`; `min_gap_s 120` / 6 per hour / 20 per day;
`avoid_unknown_place false`. A user-asked call would have gone out.
- The device was paired and the gateway (started 2026-10-23 18:54:43, carrying the memory-timeout
patch) was up throughout.
So nothing outside the agent blocked the call. The tool was simply never called.
## Root cause
**Capability asserted instead of discovered.** Both models answered from priors (“I'm a text-based
AI agent”) rather than searching the tool registry or loading the bundled skill. The first reply came
from a free model, but the mid-session switch to `smart-route` did not change the answer — so this is
not one weak model's fault. The agent treated *“I don't know of such a tool”* as *“there is no such
tool”*, in a session where the tool was one `tool_search` away.
## Evidence correction (affects the 10-22 memory-timeout incident)
`~/.hermes/conduit-calls.json` is **not** a call log:
- `history` — real call records — is **empty** (0 entries) at 2026-10-25 00:2x.
- `ends` (ring buffer, `MAX_RECENT_ENDS = 64` in `calls_store.py`) records **every turn-end that
fires a watch** (`fire()` appends `{session_id, outcome, at}`). At this audit its two entries are
**cron** sessions — `local-afac8b…` (Mnemosyne Embedding Coverage Monitor, 10-25 00:01) and
`local-8cfe41…` (Mnemosyne Health Monitor, 10-25 00:04).
Therefore the 10-24 note that *“two call sessions closed cleanly at 00:03:23 / 00:06:57”* was a
**false positive**: those timestamps match the two Mnemosyne **cron** sessions that ran that night,
not calls. Do not read `ends` as “a call happened” — read `history`. The correction is also stamped
into [[incidents/2026-10-22-conduit-memory-start-timeout]].
## Open items
- **Verify a real call end-to-end.** The memory-timeout fix has still never been exercised by a call
(`history` empty since it landed). Cheapest path: tap-to-call in the Conduit app; a chat-route call
needs the agent to actually invoke `conduit_call_user`.
- The failure is behavioural: worth checking whether the plugin's tool appears in a desktop session's
headline tool set at all (`sessions.tool_names` stores only a hash, so it could not be read from
`state.db` — the deferred-discovery path is the likely culprit for the miss).
- Consider surfacing `calling-the-user` more loudly (e.g. a system-prompt hint) so a model does not
have to discover it while a user is waiting.
## Related
- [[systems/conduit]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[concepts/tool-calling]]
+5 -4
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-24 > model routing and automation. Last updated: 2026-10-25
> >
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
@@ -32,14 +32,15 @@
- [[systems/tool-search]] — Always-on lazy discovery for external tools - [[systems/tool-search]] — Always-on lazy discovery for external tools
### Integrations ### Integrations
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, v`0.64.0` (HEAD `eaa14be`), upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device (iPhone app `2.0.0`). **Carries an uncommitted in-tree patch** — the memory-start timeout fix below. - [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.0` (HEAD `f000bf0` — 22 commits on since the 10-24 audit)**, upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`, process up since **2026-10-23 18:54:43** (so those upstream commits are **not** live); one registered device (iPhone app `2.0.0`; phone-status contact last 10-23 19:10, health ingestion to 10-24 09:59; store 569.9 MB / 1,520,229 events / 3,028 `buckets`). **Still carries the uncommitted in-tree patch — now five releases behind HEAD** (`conduit-calls.json → history` is **empty**, so the memory fix has no positive call evidence; the “two calls closed cleanly 00:03:23 / 00:06:57” reading was **wrong** — those were cron turn-ends). ⚠ **Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool** → [[incidents/2026-10-24-conduit-call-request-refused]].
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
- [[systems/scheduled-tasks]] — **As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/scheduled-tasks]] — **As of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. - [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
## Incidents ## Incidents
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it (two calls closed cleanly 00:03:23 / 00:06:57 on 10-24). **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]] - [[incidents/2026-10-24-conduit-call-request-refused]] — **The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56).** He asked to be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); all three replies asserted the agent has **no calling capability**, with **zero tool calls** in the session. `conduit_call_user` (toolset `conduit`) exists, is registered for every session, is reachable via `tool_search`, and the plugin ships the `calling-the-user` skill; his call settings (`enabled` / `when_asked` / `decides`) were permissive and the device was paired. **Root cause: capability asserted instead of discovered** — the first reply came from `kilo-auto/free`, but the switch to `smart-route` repeated it. Same audit also **corrects the 10-22 incident's closing evidence**: `conduit-calls.json → ends` records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** sessions — `history` (real calls) is **empty** and no call has been placed since the memory fix. → [[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it. ⚠ **Corrected 2026-10-25:** the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a **false positive** — those were the two Mnemosyne **cron** turn-ends of that night, and `conduit-calls.json → history` (real calls) is **empty**, so the fix has no positive call evidence and no call has been placed since. **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]] - [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery. - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]] - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
+45
View File
@@ -1154,3 +1154,48 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
syncing it, since no script owns it. syncing it, since no script owns it.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled
at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs
(25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows**
(`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) —
**nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3
`desktop`.**
- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be
phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` →
`CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session.
`conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and
has a bundled skill (`calling-the-user`); the call settings were permissive
(`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted
instead of discovered** — not a missing tool, and not one weak model (the free model's answer was
repeated after a switch to `smart-route`).
- **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence
(“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**.
`conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch**
(`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne
Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is
**empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a
call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43
restart.
- **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22
commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the
uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the
`conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified
(`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59);
store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62
`windows` was a **narrower grouping**, corrected rather than read as a collapse.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25`
section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked
superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22
line).
- **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`,
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are
the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a
`+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59
AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only.
+64 -28
View File
@@ -3,12 +3,12 @@ title: Conduit
type: system type: system
status: active status: active
created: 2026-10-24 created: 2026-10-24
updated: 2026-10-24 updated: 2026-10-25
verified_on: 2026-10-24 verified_on: 2026-10-25
review_after: 2026-11-07 review_after: 2026-11-08
confidence: high confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory] tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling]
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-log, session-20261023_182601] sources: [live-probe, plugin-tree, live-sqlite-conduit, live-sqlite-cron, live-log, session-20261024_125329]
--- ---
# Conduit # Conduit
@@ -23,12 +23,16 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
## Where it lives ## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` - **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
- **Version:** `0.64.0` (`manifest_version: 1`), git HEAD `eaa14be` — *“Pulse: mark follow-up cards - **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
done, by hand or after Do it (0.64.0) (#323)”* *“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
`0.64.0` / `eaa14be` recorded at the 2026-10-24 audit** (upstream `#324`–`#338`: Pulse
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
restart, `channels.yaml`, App Store builds following the archived plugin version).
- **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push) - **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push)
- ⚠ **The working tree carries an uncommitted local patch** (`M dashboard/plugin_api.py`, the - ⚠ **The working tree still carries the uncommitted local patch** (`M dashboard/plugin_api.py`, the
memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]). A plugin update memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]) — now **five releases
that resets the tree drops it. behind HEAD** (`0.64.0` → `0.69.0`), so the chance a plugin update resets the tree and drops it is
rising, not falling.
- **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`, - **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`,
`reading-health-data`, `using-calendar`, `using-location`. `reading-health-data`, `using-calendar`, `using-location`.
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`, - **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
@@ -36,22 +40,46 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`, installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`. `conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
## Current state (verified 2026-10-24 00:1x AWST) ## Current state (verified 2026-10-25 00:2x–00:5x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** (this - **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** and
start picked up the memory-timeout patch). **not restarted since** — so it is still the process that picked up the memory-timeout patch, and
- **Device:** one registered — **iPhone, app version `2.0.0`**, features the `0.64.0` → `0.69.0` upstream commits are **not** live.
`chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards` - **Device:** one registered — **iPhone** (`id 70aa63f3…`, `kid c24e8215…`), app version `2.0.0`,
(last seen 2026-10-23 19:02:16). No other device is registered. features `chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`;
tz `Australia/Perth`; sharing window 2026-10-16 → 2027-04-22. Paired **2026-10-22 12:54:39**.
`last_seen` / `last_sent` / `last_contact` all read **2026-10-23 19:10:30 AWST** (~29 h stale at
this audit) while health rows kept arriving until **2026-10-24 09:59:59** and window rollups were
recomputed at **13:56:35** — so those three fields track *phone-status* contact, not data flow.
(The 10-24 page recorded 19:02:16 from the same store; the epoch reads 19:10:35.)
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits - **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
- **Recent calls:** two call sessions closed cleanly on **2026-10-24 00:03:23** and - **Recent calls — none.** `conduit-calls.json → history` is **empty**: no call has been recorded
**00:06:57** (both `outcome: done`, `conduit-calls.json` written 00:06:57) — the first since the memory-timeout fix landed, so that fix still has **no positive evidence** behind it. The
evidence after the memory-timeout fix. “two call sessions closed cleanly 00:03:23 / 00:06:57” claim previously on this page was a **false
- **Data volume:** `conduit-health.sqlite3` is **561.7 MB** (WAL 6.1 MB) holding positive** — those `ends` entries are the two **cron** sessions of that night, and `ends` records
**1,501,165 `events`** rows across **308 `buckets`** / **62 `windows`** for its single device; any turn-end that fires a watch, not calls →
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **No retention or [[incidents/2026-10-24-conduit-call-request-refused]].
compaction has been observed** — the health store has grown from card/health ingestion alone - **Data volume (2026-10-25 00:2x):** `conduit-health.sqlite3` **569.9 MB** (+8.2 MB in 25.7 h),
and is worth watching against the CT460 disk budget. holding **1,520,229 `events`** (1,510,929 `sample` / 6,070 `sleep` / 3,230 `workout`),
**3,028 `buckets`** and **124 `windows`** for its single device; newest row **2026-10-24 09:59:59**,
newest `buckets.date` 2026-10-24; `conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB.
⚠ The 10-24 page recorded **308 `buckets` / 62 `windows`**; a direct `COUNT(*)` reads
**3,028 / 124**, so the earlier pair was a narrower grouping — quote these. **No retention or
compaction has been observed** — the store grows from ingestion alone and is worth watching against
the CT460 disk budget.
## Calls from Hermes (the `conduit_call_user` tool)
The plugin registers a **`conduit_call_user`** tool (toolset `conduit`, 📞) for every session and
ships the matching skill `calling-the-user`. Hermes exposes it through **lazy tool discovery**
(`tool_search`), so it is not in a session's headline tool list — it has to be looked up.
Gate conditions (`call_tool.py`): a real session (subagents are refused), a paired device,
`settings.enabled`, and either `when_asked` (with the ask recorded as user-originated) or `decides`;
the call rings when the asking turn ends and plays the reason first. Limits: `min_gap_s 120` /
6 per hour / 20 per day.
⚠ **Known failure, 2026-10-24:** three requests to be phoned were refused by the agent, which
asserted the capability did not exist **without calling the tool or loading the skill** →
[[incidents/2026-10-24-conduit-call-request-refused]].
## Memory integration ## Memory integration
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
@@ -74,17 +102,25 @@ ss -tlnp | grep 9119
# did a voice call lose its memory context? # did a voice call lose its memory context?
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
# did a real call happen? (history = calls; ends = turn-ends, NOT calls)
python3 -c "import json,os;d=json.load(open(os.path.expanduser('~/.hermes/conduit-calls.json')));print(len(d['history']),'calls;',len(d['ends']),'turn-ends')"
# state files (no secrets in the wiki) # state files (no secrets in the wiki)
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
``` ```
## Open items ## Open items
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it. - Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
- No retention policy on `conduit-health.sqlite3` (561.7 MB and growing). now five releases behind HEAD.
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* - **Verify a call end-to-end.** No call has been placed since the fix, so `MEMORY_TIMEOUT_S = 60` is
was raised. → [[systems/mnemosyne]] still unexercised by a real call, and the 10-24 refusal means the chat route has never worked →
[[incidents/2026-10-24-conduit-call-request-refused]].
- No retention policy on `conduit-health.sqlite3` (569.9 MB and growing).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
raised. → [[systems/mnemosyne]]
## Related ## Related
- [[incidents/2026-10-24-conduit-call-request-refused]]
- [[incidents/2026-10-22-conduit-memory-start-timeout]] - [[incidents/2026-10-22-conduit-memory-start-timeout]]
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
+47 -4
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-24 updated: 2026-10-25
verified_on: 2026-10-24 verified_on: 2026-10-25
review_after: 2026-10-30 review_after: 2026-10-31
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db]
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]] - **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page.
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]] - **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]]. - **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
@@ -26,6 +27,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
- **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. - **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
@@ -532,3 +534,44 @@ working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[system
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending **Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped; Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped;
the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention. the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
## Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)
**A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0.** Fleet
unchanged at **28 jobs (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident
rows**, pins unchanged. **Nothing opened and nothing closed in the incident ledger in-window.** 14
sessions: 11 `cron` + 3 `desktop`.
**The window's fault is off-cron.** On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to
test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); every reply asserted the
agent has **no calling capability**, with **zero tool calls** in the session — no `tool_search`, no
`skill_view`, no `conduit_call_user`. The tool exists, is registered for every session, is reachable
through lazy discovery, and the plugin ships the `calling-the-user` skill; his call settings
(`enabled` / `when_asked` / `decides` / `alerts`) were permissive and the device was paired. The first
reply came from `kilo-auto/free`, but the mid-session switch to `smart-route` repeated it — **capability
asserted instead of discovered**, not a broken tool →
[[incidents/2026-10-24-conduit-call-request-refused]].
**Conduit re-verified.** Plugin **`0.69.0` / HEAD `f000bf0`** (22 commits on from the `0.64.0` /
`eaa14be` recorded on 10-24) while the gateway process still dates from **2026-10-23 18:54:43**, so the
upstream commits are **not live** and the memory-timeout patch is **uncommitted and now five releases
behind HEAD**. Device phone-status contact last **10-23 19:10:30**; health ingestion ran to
**10-24 09:59:59**; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** — the 10-24
page's 308 `buckets` / 62 `windows` was a **narrower grouping**, not a collapse. ⚠ **Evidence
correction:** `conduit-calls.json → history` is **empty**, and the 10-22 incident's “two calls closed
cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** turn-ends of that night (`ends` records
any turn-end that fires a watch, `MAX_RECENT_ENDS = 64`) — so the memory fix still has **no positive
call evidence** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
**Method note:** daily tallies are bucketed with `substr(started_at,1,10)` — `date(started_at)`
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).
**Pages touched:** `incidents/2026-10-24-conduit-call-request-refused.md` (new), `systems/conduit.md`,
`incidents/2026-10-22-conduit-memory-start-timeout.md` (correction), `systems/scheduled-tasks.md`,
`current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no job retired, no system
obsolete, no decision reversed in the window.
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
health store with no retention; and a **call that has never been placed** since the memory fix.