audit 2026-10-24: conduit page + memory-start-timeout incident; 10-09 storm resolved; window 169/0; frontmatter for 2 legacy incident pages
This commit is contained in:
+38
-5
@@ -3,18 +3,51 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-23
|
||||
verified_on: 2026-10-23
|
||||
review_after: 2026-10-30
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-23
|
||||
Updated: 2026-10-24
|
||||
|
||||
## Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST)
|
||||
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST)
|
||||
|
||||
- **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23:
|
||||
**169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four;
|
||||
**2026-10-22 settled at 173 / 0**. `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
|
||||
ran clean on its 10-23 slot, closing the storm's **last two incident rows**
|
||||
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`) at **08:04** — all 12 rows closed, job `ok` again.
|
||||
**Unclosed incident rows 4 → 2**, **non-clean `last_status` 5 → 4**, **fleet unchanged at 28 jobs
|
||||
(25 enabled / 3 disabled)**, pins unchanged. → [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
||||
[[systems/scheduled-tasks]]
|
||||
- **New: Conduit is documented.** The dashboard plugin + iOS app that puts Hermes on Anthony's phone
|
||||
(voice calls, cards, push) now has a system page — plugin tree, version `0.64.0` (HEAD `eaa14be`),
|
||||
upstream `kaishi00/hermes-conduit-notifier`, gateway on `:9119`, one registered device (iPhone, app
|
||||
`2.0.0`), its state/DB files and their sizes, and the uncommitted patch it carries.
|
||||
→ [[systems/conduit]]
|
||||
- **⚠ New fault, found and fixed in-window: Conduit voice calls were being abandoned by the memory
|
||||
layer.** The plugin hard-coded a **10 s** budget for one request *including the Mnemosyne provider
|
||||
start*, while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls hit it
|
||||
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed
|
||||
out; abandoning the call`) and ran **without recall**. Patched in the plugin's working tree to a **60 s**
|
||||
default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that proceeds without recall
|
||||
instead of aborting; the dashboard restarted **18:54:43** and **no abort line exists after it**, with two
|
||||
calls closing cleanly at 00:03:23 and 00:06:57 on 10-24. **The patch is uncommitted** — a plugin update
|
||||
reverts it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/mnemosyne]]
|
||||
- **Sessions: 14 in the window** — 13 `cron` + 1 `desktop` (10-23 18:26:12, the phone-call session above);
|
||||
no other interactive sessions.
|
||||
- **Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
|
||||
(`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||
- **Housekeeping:** frontmatter added to `incidents/2026-07-29-mnemosyne-relay.md` and
|
||||
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md`, both of which sat outside the SCHEMA
|
||||
(rule 4). Freshness: 102 pages; **3** carry a `verified_on` older than 21 days — `infrastructure/hosts.md`,
|
||||
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` (all 94 d). Nothing removed or archived.
|
||||
|
||||
## Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST) *(superseded by the 2026-10-24 entry above)*
|
||||
|
||||
- **First zero-failure cron day in three days, and five incidents cleared at once.** Window 2026-10-22: **172
|
||||
`completed` / 0 `failed`** (+ this audit `running`); 4 runs had already completed on 10-23 by 00:0x. **Five
|
||||
|
||||
@@ -1,3 +1,16 @@
|
||||
---
|
||||
title: Mnemosyne Relay Scope Corruption + `tools: None` — 2026-07-29
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-07-29
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [mnemosyne, config, relay, incident, resolved]
|
||||
sources: [live-probe, config, session-20260729]
|
||||
---
|
||||
|
||||
# mnemosyne-tools-none — 2026-07-29
|
||||
|
||||
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
|
||||
|
||||
@@ -1,3 +1,16 @@
|
||||
---
|
||||
title: OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04
|
||||
type: incident
|
||||
status: monitoring
|
||||
created: 2026-08-04
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [omniroute, deepseek, free-tier, silent-failure, incident, mitigated]
|
||||
sources: [live-probe, omniroute-log, session-20260804]
|
||||
---
|
||||
|
||||
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
|
||||
|
||||
## Status
|
||||
|
||||
@@ -1,22 +1,23 @@
|
||||
---
|
||||
title: Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
type: incident
|
||||
status: unresolved
|
||||
status: resolved
|
||||
created: 2026-10-09
|
||||
updated: 2026-10-13
|
||||
verified_on: 2026-10-13
|
||||
review_after: 2026-10-16
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [cron, omniroute, router, timeout, incident, unresolved]
|
||||
tags: [cron, omniroute, router, timeout, incident, resolved]
|
||||
sources: [cron-executions-db, cron-incidents-db, live-jobs-json, session-20261009_225935_7b1cb1]
|
||||
---
|
||||
|
||||
# Fleet-Wide Provider Timeout Storm — 2026-10-09
|
||||
|
||||
**Window:** 2026-10-09 00:00 → 15:00 AWST (last failure 15:00; every run from 18:00 onward completed)
|
||||
**Status:** Unresolved — the storm ended and **11 of its 12 incidents are now closed**; only
|
||||
`1b23c0_c35703f637b8` remains open, and it cannot clear before that job's next scheduled slot
|
||||
(**Fri 2026-10-16 08:00**). Nothing here is still symptomatic.
|
||||
**Status:** Resolved — **all 12 of its incident rows are closed.** The last hold-outs
|
||||
(`1b23c0_c35703f637b8` and `1b23c0_103cbe04a79b`, both on `1b23c042e17d`, the weekly Friday job)
|
||||
closed **2026-10-23 08:04** when that job's Friday slot ran clean; its `last_status` is `ok` again.
|
||||
The storm contributes **no** unclosed rows. Nothing here is still symptomatic.
|
||||
**Impact:** 15 failed executions across 10 distinct jobs, against baselines of **1 failed on 2026-10-07** and **2 on 2026-10-08**
|
||||
|
||||
## What happened
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: Conduit Voice Call Loses Its Memory Context (Mnemosyne Start Timeout)
|
||||
type: incident
|
||||
status: resolved
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [conduit, mnemosyne, timeout, plugin, voice, incident, resolved]
|
||||
sources: [live-log, plugin-tree, live-sqlite-conduit, git-diff]
|
||||
---
|
||||
|
||||
# Conduit Voice Call Loses Its Memory Context — 2026-10-22 → 2026-10-23
|
||||
|
||||
**Window:** 2026-10-22 11:28 → 2026-10-23 18:29 AWST (three occurrences)
|
||||
**Status:** Resolved 2026-10-23 18:54 (patched and verified)
|
||||
**Impact:** A Conduit voice call that arrived before the Mnemosyne provider was warm lost its
|
||||
memory context — and, in the pre-fix code path, the *call itself* was abandoned by the plugin's
|
||||
memory layer.
|
||||
|
||||
## Symptom
|
||||
|
||||
Three identical warnings in `~/.hermes/logs/agent.log` (also in `errors.log`), one per call attempt
|
||||
that hit a cold memory provider:
|
||||
|
||||
```
|
||||
2026-10-22 11:28:17,137 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,867 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-23 18:29:16,830 WARNING hermes_dashboard_plugin_conduit_push: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call
|
||||
2026-10-22 13:16:37,875 WARNING hermes_dashboard_plugin_conduit_push: Memory context for Conduit failed: Memory start timed out
|
||||
```
|
||||
|
||||
## Root cause
|
||||
|
||||
`dashboard/plugin_api.py` in the `conduit_push` plugin hard-coded `MEMORY_TIMEOUT_S = 10.0` — a
|
||||
budget for **one request including the provider start**. Mnemosyne's cold start for the
|
||||
`conduit-voice` session takes roughly **10–45 s**, so any call landing on a cold provider blew the
|
||||
budget. The plugin then ran the call **without memory context**, and the failure was logged as
|
||||
`Memory start timed out` — the agent speaks to the user with no recall of prior sessions and no
|
||||
error surfaced to the caller.
|
||||
|
||||
This is not an OmniRoute/model fault, and not a Mnemosyne data fault: the store was healthy and
|
||||
the provider simply needed longer than the plugin allowed. → [[systems/mnemosyne]]
|
||||
|
||||
## Fix applied (2026-10-23, live before 18:54:43)
|
||||
|
||||
A single-file change to `~/.hermes/plugins/conduit_push/dashboard/plugin_api.py`, **still
|
||||
uncommitted in the working tree**:
|
||||
|
||||
```diff
|
||||
-MEMORY_TIMEOUT_S = 10.0
|
||||
+# Mnemosyne (the memory provider) can take ~10-45s to cold-start for the
|
||||
+# conduit-voice session, so give it enough room; tunable via env.
|
||||
+MEMORY_TIMEOUT_S = float(os.environ.get("CONDUIT_MEMORY_TIMEOUT_S", "60.0"))
|
||||
```
|
||||
|
||||
and a hard-failure guard so a token/start error degrades instead of aborting:
|
||||
|
||||
```diff
|
||||
- entry = _memory_providers.get(key, name, deadline)
|
||||
+ try:
|
||||
+ entry = _memory_providers.get(key, name, deadline)
|
||||
+ except TokenError:
|
||||
+ logger.warning("Memory provider %r failed to start for Conduit; proceeding without recall", name)
|
||||
+ entry = None
|
||||
```
|
||||
|
||||
The new default is used by every deadline in the memory layer (6 call sites) plus the
|
||||
`+ 2.0 s` backstop around each provider call. `CONDUIT_MEMORY_TIMEOUT_S` overrides it without
|
||||
touching the file.
|
||||
|
||||
## Verification
|
||||
|
||||
- Dashboard process on `:9119` restarted **2026-10-23 18:54:43** — after the 18:29 abort and
|
||||
therefore running the patched code.
|
||||
- **Zero** `abandoning the call` / `Memory context for Conduit failed` lines exist after that
|
||||
restart (`grep` across `~/.hermes/logs/*.log`, deduped).
|
||||
- Two Conduit call sessions closed cleanly after the restart — **2026-10-24 00:03:23** and
|
||||
**00:06:57**, both `outcome: done`.
|
||||
- **Caveat, stated plainly:** the plugin logs only *failures*, so the absence of a warning is
|
||||
not positive proof that recall was populated on those calls. The proven part is that the call
|
||||
is no longer abandoned by the memory layer.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- **The patch is uncommitted.** A plugin update that resets
|
||||
`~/.hermes/plugins/conduit_push` reverts it to the 10 s budget. Commit it or send it upstream
|
||||
(`kaishi00/hermes-conduit-notifier`).
|
||||
- The underlying Mnemosyne cold start (10–45 s) is unchanged — only the budget was raised.
|
||||
Consumers with tight timeouts elsewhere will hit the same wall. → [[systems/mnemosyne]]
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]]
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-22
|
||||
> model routing and automation. Last updated: 2026-10-24
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
@@ -32,20 +32,22 @@
|
||||
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||
|
||||
### Integrations
|
||||
- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, v`0.64.0` (HEAD `eaa14be`), upstream `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device (iPhone app `2.0.0`). **Carries an uncommitted in-tree patch** — the memory-start timeout fix below.
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
|
||||
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||
- [[systems/scheduled-tasks]] — **As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**306 skills / 70 categories** as of the **2026-10-23 03:07** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at that run: **2 stale / 94 fresh / 6 no-date of 102 pages** (`freshness-20261023.txt`). Note: `SKILL.md` files on disk still exceed the index (the generator de-dups one nested duplicate; see [[systems/scheduled-tasks]]). ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole.
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` **`alerted`, unclosed**; the only cron failure of the day (171 `completed` / 1 `failed`). **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]] — **Conduit voice calls were silently running without memory, 2026-10-22 → 10-23.** The phone plugin gave one request — *including the Mnemosyne provider start* — a hard-coded **10 s** budget (`MEMORY_TIMEOUT_S`), while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls were abandoned at the memory layer (`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with **no recall of prior conversation**. Fixed 10-23 in the plugin's working tree: default **60 s** (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that logs and proceeds without recall instead of aborting; dashboard restarted **18:54:43** and no abort line exists after it (two calls closed cleanly 00:03:23 / 00:06:57 on 10-24). **The patch is uncommitted — a plugin update reverts it.** The cold start itself was **not** optimised. → [[systems/conduit]], [[systems/mnemosyne]]
|
||||
- [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] — **A third failure signature from the `custom:omni-101` / `smart-route` pin, 2026-10-21.** The Daily Marketing Job Radar (`dca8482e4f76`, 07:00 weekday) died at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **no 503, no connection error, no repetition loop**, and **zero 503 rows exist for that day** (last 2026-10-19 09:02). Its 07:00 sibling `85edf6772bca` completed normally, so the node was serving; the run wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls). Incident `dca848_014affeb6954` was **closed 2026-10-22 07:59:22** when the next daily slot ran clean. **Not** caused by the 22:47 `hermes update` — that landed 15 h later. → [[systems/omniroute]]
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — **The gateway exited uncleanly on 2026-10-20 (09:09:01, `suspected_oom=False`) and orphaned a live cron run.** `08eb86f02d37` (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed **`unknown`** — *“Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.”* **No delivery row exists for the slot and no incident row was created**; the job table keeps showing the *previous* slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a **27-day clean run**. Open: orphan-recovery on restart, and treating a delivery-targeted `unknown` as a probable non-delivery.
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — **Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47.** `e27da6c34bbb` reads `last_status: ok` and its executions read `completed` every night while **75 commits sit unpushed** (`fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'` — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (`message_id 8447`, 2026-10-20 11:00); AgentMail failed. **No off-box `~/.hermes` backup in 2.5 months.** Second instance of the pattern after the GitHub Trending Radar token revocation. → [[infrastructure/backups]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? **Second surface (2026-10-20 18:37):** an interactive `desktop` session received the same canned line after `Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … response_len=0`, so the substitute is emitted by the harness's failure path **generally**, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — **was never answered**.
|
||||
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status: `active` → back to `monitoring` on 2026-10-21** — 2026-10-20 ran **168 / 0** with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`); **unclosed rows 13 → 9** (8 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.**
|
||||
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST. **CLOSED — all 12 incident rows are now resolved**; the last two (`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on the weekly Friday job `1b23c042e17d`) closed **2026-10-23 08:04** when that slot ran clean, and the job reads `ok` again
|
||||
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
||||
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
|
||||
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
||||
|
||||
@@ -1107,3 +1107,50 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st
|
||||
`systems/model-providers.md` (22 d, updated 2026-10-01). The latter needs a review pass against the live
|
||||
OmniRoute catalogue before its next `verified_on`.
|
||||
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
|
||||
|
||||
## [2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST
|
||||
|
||||
- **New page:** `systems/conduit.md` — the Conduit dashboard plugin + iOS app (voice calls, cards,
|
||||
push). Plugin tree `~/.hermes/plugins/conduit_push`, version `0.64.0` (HEAD `eaa14be`, no tags
|
||||
upstream), remote `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device
|
||||
(iPhone, app `2.0.0`); state/DB files and sizes recorded. Where the wiki previously had no page for
|
||||
the phone surface at all.
|
||||
- **New incident:** [[incidents/2026-10-22-conduit-memory-start-timeout]] — three
|
||||
`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call` warnings
|
||||
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s `MEMORY_TIMEOUT_S` in the
|
||||
plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree
|
||||
to a 60 s default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades
|
||||
instead of aborting; the dashboard restarted 18:54:43 and **no abort line exists after it** (two
|
||||
calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). **The patch is uncommitted.**
|
||||
- **`incidents/2026-10-09-cron-provider-timeout-storm.md` → `status: resolved`.** Its last two rows
|
||||
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on `1b23c042e17d`) closed **2026-10-23 08:04**
|
||||
when that weekly job's Friday slot ran clean; the job reads `ok` again. **All 12 rows closed** —
|
||||
this incident no longer contributes an unclosed row anywhere.
|
||||
- **Cron ledger:** window **2026-10-23 closed 169 `completed` / 0 `failed`** (+ this audit `running`) —
|
||||
the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. Fleet **28 jobs (25 enabled /
|
||||
3 disabled)**, unchanged for a sixth audit. **Unclosed incident rows 4 → 2** (`c0a9fa_764558b5ae83`
|
||||
`alerted` since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and
|
||||
`0e753e_e7f8194a7d06` `detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4**
|
||||
(`c0a9fa314542`, `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Pins unchanged (11 `omni-101` /
|
||||
`smart-route`, 8 unpinned, 6 `omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1
|
||||
`deepseek-paid-smart` = this audit). **14 sessions:** 13 `cron` + 1 `desktop` (18:26:12, the Conduit
|
||||
phone-call session that exposed the fault above).
|
||||
- **`systems/mnemosyne.md`:** new block recording the 10–45 s cold start and the consumer-timeout rule
|
||||
(a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold
|
||||
start itself was **not** optimised — only the budget was raised.
|
||||
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-24`
|
||||
section), `current-state.md` (new Recent Changes block; 10-23 block marked superseded), `index.md`
|
||||
(date, new `systems/conduit` and incident lines, refresh of the scheduled-tasks / skills-index /
|
||||
storm blurbs), and **frontmatter added** to `incidents/2026-07-29-mnemosyne-relay.md` and
|
||||
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md` — both sat outside the SCHEMA
|
||||
(rule 4: every page has frontmatter).
|
||||
- **Freshness:** 102 pages scanned at the last stocktake; **3 carry a `verified_on` older than 21 days**
|
||||
— `infrastructure/hosts.md`, `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`
|
||||
(all 94 d, unchanged). **Nothing removed or archived** this cycle: no job retired, no system
|
||||
obsolete, no decision reversed in the window.
|
||||
- **Note (hygiene):** a second, older copy of this wiki sits at `/home/hermes/wiki/agent-estate-wiki`
|
||||
(101 pages, 3 incidents, no git remote, last commit `c798304` "mirror(2026-10-22 maintenance)").
|
||||
It is **not** served — `hermes-wiki-serve.service` serves `public/` from *this* repo — so it is a
|
||||
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
|
||||
syncing it, since no script owns it.
|
||||
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
title: Conduit
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-10-24
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [conduit, plugin, phone, iphone, push, voice, memory]
|
||||
sources: [live-probe, plugin-tree, live-sqlite-conduit, live-log, session-20261023_182601]
|
||||
---
|
||||
|
||||
# Conduit
|
||||
|
||||
## Purpose
|
||||
Hermes on Anthony's phone. A dashboard plugin + iOS app pair that pushes cards, prompts,
|
||||
location and health data between the Hermes gateway and the iPhone, and lets a **voice call**
|
||||
be started from the phone where the agent can speak and listen (`conduit-voice` sessions).
|
||||
|
||||
Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[systems/hermes-ui]]
|
||||
(the browser desktop), [[systems/push-notifications]] (the older `hermes-push` bridge).
|
||||
|
||||
## Where it lives
|
||||
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
|
||||
- **Version:** `0.64.0` (`manifest_version: 1`), git HEAD `eaa14be` — *“Pulse: mark follow-up cards
|
||||
done, by hand or after Do it (0.64.0) (#323)”*
|
||||
- **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push)
|
||||
- ⚠ **The working tree carries an uncommitted local patch** (`M dashboard/plugin_api.py`, the
|
||||
memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]). A plugin update
|
||||
that resets the tree drops it.
|
||||
- **Skills shipped with the plugin** (4, under `plugins/conduit_push/skills/`): `calling-the-user`,
|
||||
`reading-health-data`, `using-calendar`, `using-location`.
|
||||
- **Data stores** (`~/.hermes/`): `conduit-health.sqlite3`, `conduit-location.sqlite3`,
|
||||
`conduit-phone.sqlite3`, plus JSON state — `conduit-push.json` (credential / gateway id /
|
||||
installation id / relay url / e2e — **no values recorded here**), `conduit-calls.json`,
|
||||
`conduit-settings.json`, `conduit-present.json`, `conduit-desktop-views.json`.
|
||||
|
||||
## Current state (verified 2026-10-24 00:1x AWST)
|
||||
- **Dashboard gateway:** listening on `0.0.0.0:9119`; process started **2026-10-23 18:54:43** (this
|
||||
start picked up the memory-timeout patch).
|
||||
- **Device:** one registered — **iPhone, app version `2.0.0`**, features
|
||||
`chat-cards`, `info-cards`, `live-cards`, `maps-navigate`, `media-cards`, `show-cards`
|
||||
(last seen 2026-10-23 19:02:16). No other device is registered.
|
||||
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
|
||||
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
|
||||
- **Recent calls:** two call sessions closed cleanly on **2026-10-24 00:03:23** and
|
||||
**00:06:57** (both `outcome: done`, `conduit-calls.json` written 00:06:57) — the first
|
||||
evidence after the memory-timeout fix.
|
||||
- **Data volume:** `conduit-health.sqlite3` is **561.7 MB** (WAL 6.1 MB) holding
|
||||
**1,501,165 `events`** rows across **308 `buckets`** / **62 `windows`** for its single device;
|
||||
`conduit-location.sqlite3` 80 KB; `conduit-phone.sqlite3` 284 KB. **No retention or
|
||||
compaction has been observed** — the health store has grown from card/health ingestion alone
|
||||
and is worth watching against the CT460 disk budget.
|
||||
|
||||
## Memory integration
|
||||
The voice session (`conduit-voice`) reaches Mnemosyne through the plugin's own memory layer, not
|
||||
through Mnemosyne's `system_prompt_block()` — recall is wrapped in a per-request deadline
|
||||
(`MEMORY_TIMEOUT_S`, default **60 s** as of 2026-10-23, overridable with
|
||||
`CONDUIT_MEMORY_TIMEOUT_S`; it was a hard-coded **10 s** before that, which was too short for
|
||||
Mnemosyne's 10–45 s cold start). If the provider fails to start, the plugin logs a warning and
|
||||
proceeds without recall rather than abandoning the call.
|
||||
See [[incidents/2026-10-22-conduit-memory-start-timeout]] and [[systems/mnemosyne]].
|
||||
|
||||
## Maintenance commands
|
||||
```bash
|
||||
# what version is installed / is the tree patched?
|
||||
git -C ~/.hermes/plugins/conduit_push log --oneline -1 && \
|
||||
git -C ~/.hermes/plugins/conduit_push status --short
|
||||
|
||||
# is the gateway up, and since when?
|
||||
ss -tlnp | grep 9119
|
||||
|
||||
# did a voice call lose its memory context?
|
||||
grep -h "Conduit" ~/.hermes/logs/agent.log | grep -i "memory" | tail
|
||||
|
||||
# state files (no secrets in the wiki)
|
||||
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
|
||||
```
|
||||
|
||||
## Open items
|
||||
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it.
|
||||
- No retention policy on `conduit-health.sqlite3` (561.7 MB and growing).
|
||||
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget*
|
||||
was raised. → [[systems/mnemosyne]]
|
||||
|
||||
## Related
|
||||
- [[incidents/2026-10-22-conduit-memory-start-timeout]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[current-state]]
|
||||
+20
-2
@@ -3,8 +3,9 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-11-07
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -157,3 +158,20 @@ Primary agent memory store for Hermes.
|
||||
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e.
|
||||
one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on
|
||||
every slot since (latest 10-18 00:30).
|
||||
|
||||
## Re-verified 2026-10-24 (audit window 2026-10-23; live probes 2026-10-24 00:1x–00:3x AWST)
|
||||
|
||||
- **Cold start is the new failure vector for consumers: 10–45 s.** The Conduit voice plugin
|
||||
(`conduit_push`) gave one request — provider start included — a hard-coded **10 s** budget
|
||||
(`MEMORY_TIMEOUT_S`). Mnemosyne's cold start for the `conduit-voice` session regularly exceeds
|
||||
that, so **three calls were abandoned by the memory layer** (10-22 11:28:17, 10-22 13:16:37,
|
||||
10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call`),
|
||||
and the caller got an agent with no recall. Fixed in the plugin's working tree on 10-23: default
|
||||
**60 s**, overridable with `CONDUIT_MEMORY_TIMEOUT_S`, plus a `TokenError` guard that logs and
|
||||
proceeds without recall instead of aborting. Dashboard restarted 18:54:43; no abort line exists
|
||||
after it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]
|
||||
- **Rule for other consumers:** a 10 s memory budget is below the observed cold-start floor. Any
|
||||
new surface that reads Mnemosyne on a first request needs ≥60 s, or must degrade to "no recall"
|
||||
rather than fail. The **underlying cold start was not optimised** — only the budget was raised.
|
||||
- **Capture and coverage unchanged** from the 10-18 read (working 760/786 = 96.7 %, episodic
|
||||
1050/1056 = 99.4 %; nothing indicates a reindex is due). No monitor reds in the window.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-23
|
||||
verified_on: 2026-10-23
|
||||
updated: 2026-10-24
|
||||
verified_on: 2026-10-24
|
||||
review_after: 2026-10-30
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-23 00:1x AWST (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||
- **Current as of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||||
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||||
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
@@ -486,3 +487,48 @@ OmniRoute node — the durable fix is a second provider pin or a router-side fai
|
||||
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
|
||||
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
|
||||
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.
|
||||
|
||||
## Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x)
|
||||
|
||||
**Count.** As of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`): **28 jobs —
|
||||
25 enabled / 3 disabled**, unchanged for a sixth consecutive audit. Window **2026-10-23: 169 `completed` /
|
||||
0 `failed`** (+ this audit `running`), the third zero-failure day in four; **2026-10-22 settled at 173 / 0**.
|
||||
|
||||
**The 2026-10-09 timeout storm closed.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
|
||||
ran clean on its 10-23 slot, closing the storm's last two rows (`1b23c0_c35703f637b8`,
|
||||
`1b23c0_103cbe04a79b`) at **08:04**. The storm now holds **zero** unclosed rows and the job reads `ok`
|
||||
again → [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||||
|
||||
**Incident ledger: unclosed 4 → 2** (no rows opened, two closed). Remaining: `c0a9fa_764558b5ae83`
|
||||
(`alerted` since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and
|
||||
`0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01 — Facebook Birthday monthly scrape).
|
||||
|
||||
**Non-clean `last_status`: 5 → 4** — `c0a9fa314542` (`error`, streak 1, Mon 10-26), `0e753e9720fd`
|
||||
(`error`, 0, 11-01), `fad5a86979d4` (`delivery_failed`, Mon 10-26), `066bb5191280` (`delivery_failed`, 11-01).
|
||||
No new failure signature appeared in the window.
|
||||
|
||||
**Pins unchanged** — 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6
|
||||
`custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` /
|
||||
`deepseek-paid-smart` (this audit).
|
||||
|
||||
**Sessions: 14 in the window** — 13 `cron` + **1 `desktop`** (2026-10-23 18:26:12, 13 messages, the Conduit
|
||||
phone-call session that exposed the memory-timeout fault below).
|
||||
|
||||
**New fault found and fixed in-window (not a cron job):** the **Conduit** voice plugin gave the Mnemosyne
|
||||
provider start a hard-coded **10 s** budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16)
|
||||
were abandoned by the memory layer and ran without recall. Patched the same evening to a **60 s** default
|
||||
(`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades instead of aborting; dashboard
|
||||
restarted 18:54:43 and **no abort line exists after it**. The patch is **uncommitted** in the plugin's
|
||||
working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
|
||||
|
||||
**Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
|
||||
(`250c735`: freshness 2 stale / 94 fresh, skills-index 306).
|
||||
|
||||
**Pages touched:** `systems/conduit.md` (new), `incidents/2026-10-22-conduit-memory-start-timeout.md` (new),
|
||||
`incidents/2026-10-09-cron-provider-timeout-storm.md` (status → resolved), `systems/mnemosyne.md`,
|
||||
`current-state.md`, `index.md`, `log.md`, plus frontmatter added to two legacy incident pages (see log).
|
||||
**Nothing removed or archived:** no job retired, no system obsolete in the window.
|
||||
|
||||
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
|
||||
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped;
|
||||
the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
|
||||
|
||||
Reference in New Issue
Block a user