diff --git a/incidents/2026-10-31-asana-access-failures.md b/incidents/2026-10-31-asana-access-failures.md new file mode 100644 index 0000000..eaf3cf4 --- /dev/null +++ b/incidents/2026-10-31-asana-access-failures.md @@ -0,0 +1,102 @@ +--- +title: Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape +type: incident +status: active +created: 2026-11-01 +updated: 2026-11-01 +verified_on: 2026-11-01 +review_after: 2026-11-08 +confidence: high +tags: [hermes, automation, mcp, connector, credential, silent-failure] +sources: [live-mcp-list, live-config-yaml, live-mcp-tokens, live-logs, live-probe] +--- + +# Asana Is Unreachable From CT460 on Both Paths — MCP Server Never Authenticated, Hosted Connector Rejects the Call Shape + +## Headline + +The estate has **two independent Asana integrations on CT460, and neither can complete a read** — which +went unnoticed until an interactive session spent 2026-10-31 evening retrying both to answer a single +question ("where does Remembrance Day stand?"). + +| Path | Surface | Live state 2026-10-31/11-01 | +|---|---|---| +| **MCP server `asana`** | `https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true` (`hermes mcp list` → ✓ enabled) | **Never authenticated.** Every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first to complete initial authorization.` | +| **Hosted connector** | `connectors__asana__GET_A_TASK`, `…__SEARCH_TASKS_IN_WORKSPACE` | Reaches Asana but returns **`422 INVALID_ARGUMENTS` — "Input should be a valid list on parameter \`opt_fields\`"**. 12+ identical retries in one session. | + +## Evidence + +**MCP server — no token, ever.** `~/.hermes/mcp-tokens/` holds exactly one file: + +``` +-rw------- 1 hermes hermes 319 Oct 26 01:07 asana.client.json +``` + +A registered **client** with **no token file** → the OAuth flow was started (client registered +**2026-10-26 01:07**) and never completed. `config.yaml → mcp_servers.asana` reads +`{auth: oauth, enabled: true, url: https://mcp.asana.com/v2/mcp}`. The earliest Asana trace in retained +logs is an **HTTP 401** from `mcp.asana.com/v2/mcp` at **2026-10-27 16:35:05**, followed immediately by +OAuth discovery calls. + +**Failure volume** (log lines across `logs/agent.log*` + `logs/errors.log*`; the failure is written to +both files, so line counts roughly double the number of events): + +| Day | `MCP OAuth setup failed for 'asana'` lines | +|---|---| +| 2026-10-21 | 455 | +| 2026-10-22 | 469 | +| 2026-10-23 | 576 | +| 2026-10-24 | 576 | +| 2026-10-25 | 576 | +| 2026-10-26 | 301 | +| 2026-10-27 | 439 | +| 2026-10-28 | 1 150 | +| 2026-10-29 | 1 154 | +| 2026-10-30 | 862 | +| **2026-10-31** | **1 156** | +| 2026-11-01 (to 00:16) | 14 | +| **Total retained** | **7 728** | + +The earliest retained line is **2026-10-21 05:09:43** (`errors.log.2`) — the gap **predates the retained +log window**; this is a chronic condition, not a 10-31 regression. The 10-31 peak coincides with the +evening Pulse session hammering the path. + +**Hosted connector — the call, not the credential.** From `20261031_210329_b35440` (the Conduit Pulse +"Remembrance Day" session), repeated at 22:17:47, 22:20:42, 22:26:31, 22:31:25, 22:36:45, 23:14:45, +23:26:46, 23:28:55, 23:30:34, 23:39:31, 23:41:48, 23:51:19 — each call burning 11–20 s: + +``` +tools.managed_gateway_identity: connector connectors.execute 422 INVALID_ARGUMENTS +Tool tool_call returned error: {"results":[{"name":"connectors__asana__GET_A_TASK","error": + {"code":"INVALID_ARGUMENTS","message":"Invalid request data provided\n- Input should be a valid list + on parameter `opt_fields`…"}}]} +``` + +The connector is **authorised** (it is the MCP path that cannot authenticate), so the block is purely +argument shape: `opt_fields` must be a **JSON list**, and a string/comma-form is rejected. + +## Impact + +- **Non-interactive sessions (cron, background jobs, Conduit background investigations) cannot read + Asana at all** — the MCP path needs an interactive browser login that cron cannot provide. +- **Interactive sessions** get the hosted connector, but the agent re-issued the same mis-shaped call + a dozen times instead of adapting — ~3 minutes of model time spent failing identically. +- Asana-backed user requests (task status behind the Conduit Pulse flow) degrade to guesses. + +## Remediation (not performed — this audit records, it does not re-authorise) + +1. Run **`hermes mcp login asana`** from an interactive session to complete the OAuth flow; confirm + `~/.hermes/mcp-tokens/asana.json` (or equivalent token file) appears afterwards, then re-probe the + MCP server. +2. **Or** disable the `asana` MCP server if the hosted connector is the intended path — two Asana paths + on one host is itself the source of the confusion, and one of them has never worked. +3. For the connector: pass **`opt_fields` as a JSON list**, not a comma-separated string. +4. Watch for the silent-cost pattern: a path that fails **every** time still burns a tool call each + time. 1 100+ failures/day means the failure is being retried, not fixed. + +## Related + +- [[systems/hermes-agent]] — MCP server inventory and auth state +- [[systems/conduit]] — the Pulse flow that needed the Asana task status +- [[systems/scheduled-tasks]] — cron fleet (the surface that cannot authenticate interactively) +- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the same "green row, dead destination" class diff --git a/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md b/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md new file mode 100644 index 0000000..fcdf8d0 --- /dev/null +++ b/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md @@ -0,0 +1,103 @@ +--- +title: Conduit "voice mode: local" never applies — Mac mini speech server's LLM proxy is disabled +type: incident +status: active +created: 2026-11-01 +updated: 2026-11-01 +verified_on: 2026-11-01 +confidence: high +tags: [conduit, voice, local-live, mac-mini, speech-server, investigation] +sources: + - session 20261031_193709_40b9c2 (Conduit voice session, started 2026-10-31 19:37:09 AWST) + - ~/.hermes/conduit-settings.json, ~/.hermes/conduit-calls.json, ~/.hermes/config.yaml + - live probes of the Mac mini speech server (LLM endpoint, /v1/usage, realtime socket) +--- + +# Conduit "voice mode: local" never applies + +**Status: OPEN — investigation in progress. Nothing was changed on the Mac mini speech server.** + +On 2026-10-31 Anthony reported (from the phone): switching the Conduit app's voice +setting to **"local"** shows a **successful** connection test, but the change never +takes effect. A background job launched from that voice conversation +(`20261031_193709_40b9c2`, started **19:37:09 AWST**, 30 messages / 17 tool calls) was +still open at the 2026-11-01 00:2x audit. + +## Why the test succeeds and the change still does nothing + +The "test connection" button checks **socket reachability only** — it never exercises +the model path. So it passes while every local turn is still impossible: + +| # | Finding | Evidence | +|---|---------|----------| +| 1 | The Mac mini speech server's **LLM proxy is disabled** | its LLM endpoint answers **HTTP 501** — *"The LLM proxy is disabled. Start the server with `--enable_llm_proxy` to enable it."* | +| 2 | The Mac has never served a local turn | `/v1/usage` → **`llm_proxy.requests: 0`** across **25** observed turns | +| 3 | The realtime handshake is not understood | the Mac's realtime socket **accepts the connection** but rejects **`session.update`** as an **unknown event** | +| 4 | The app's test button proves nothing about the model path | it returns success on socket reachability alone | + +Findings 1–3 are the strongest single explanation: **even with `voice.mode = local` +set on the phone, the Mac cannot call a model back.** Finding 4 is why the UI reports +success. + +## `voice.mode` is phone-side, not host-side + +The host never reads a `voice.mode` key — there is none to read: + +- `~/.hermes/config.yaml → voice` holds **`auto_tts`**, **`local_live`**, + **`speech_server`** — **no `mode`**. +- `~/.hermes/conduit-settings.json` contains **no `voice.*` keys at all**; its only + device state is `devices//mode: small` for three devices. +- The device itself reports **`voice.mode: "classic"`**. + +So "set mode to local and the change does not apply" cannot be diagnosed on the host +config — the selector lives in the app, and what the host does with it is a +**`local_live`** path decision. + +## A misread that was caught mid-session (worth keeping) + +`local_live.py:882` is the **only** writer of `_local_live_last_request`, and it sits +inside the **chat-completions route**. A "recent `last_request_at`" reading during the +investigation therefore indicated **this investigation's own probe**, not phone +traffic — the agent initially read it as phone activity and corrected the call in the +same session. Live check: `conduit-calls.json → last_request_at` is **`null`**. + +Lesson: on this integration, `last_request_at` is proof of *some* HTTP request hitting +the local-live route, **not** proof of a phone call. + +## Configuration at the time of writing + +Changed during the 2026-10-31 session (`config.yaml` mtime **21:02:30**): + +- `voice.local_live.front_model` = `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}` + — was `mistral/ministral-14b-latest`. + +Unchanged: `voice.speech_server.url = wss://…:8443/v1/realtime`, `voice.auto_tts: false`, +`tts.provider: ollama_local`, `stt.provider: local`. + +The Conduit dashboard gateway was restarted at **19:35:02** (pid 176771), minutes +before the session opened. + +## Prompting context (candidate, not proven) + +The plugin **self-updated to `0.122.2` on 2026-10-31 18:05**, which drove a gateway +reload at **18:06:41** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. +A newly shipped local-live path is a reasonable regression candidate, but the +disabled LLM proxy would break local mode on **any** plugin version, so the version +change is not yet implicated. + +## Open items + +- **Enable the Mac mini's LLM proxy** (`--enable_llm_proxy`) or confirm the intended + Local Live architecture — with the proxy off, finding 1 alone explains the symptom. +- Confirm the **realtime event vocabulary** the plugin expects vs. what the Mac's + socket implements (`session.update` is currently rejected as unknown). +- Decide whether `front_model` should be a **text** or **audio** model — a + chat-completions `base_url` pointed at an audio model may itself be the mismatch. +- Re-test from the phone after any change; the app's own test button is not evidence. + +## Related + +- [[systems/conduit]] +- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] +- [[incidents/2026-10-29-conduit-voice-silence]] +- [[incidents/2026-10-22-conduit-memory-start-timeout]] diff --git a/incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md b/incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md new file mode 100644 index 0000000..10658b1 --- /dev/null +++ b/incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md @@ -0,0 +1,101 @@ +--- +title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in) +type: incident +status: active +created: 2026-11-01 +updated: 2026-11-01 +verified_on: 2026-11-01 +review_after: 2026-11-08 +confidence: high +tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure] +sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe] +--- + +# Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in) + +## Headline + +On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` / +`hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the +whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and +force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), +whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on +**`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**. + +This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the +2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain +mechanism behaved as designed and still ended in a lost run. + +## Timeline (live gateway log, `~/.hermes/logs/agent.log`) + +| Time (AWST) | Event | +|---|---| +| 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` | +| 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` | +| 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` | +| 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit | +| 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred | +| 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` | +| **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) | +| 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` | +| 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) | +| 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) | +| 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s | +| 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot | +| 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` | + +Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the +2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and +`0c771463c3cd` slot from 19:15:21 onward `completed`). + +## Incident-ledger effect + +| Row | Job | State | First seen | Last seen | +|---|---|---|---|---| +| `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 | +| `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 | +| `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 | + +The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now +describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"* +(`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.** + +## What is genuinely new here + +1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26 + incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit + (`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that + overlaps a long-running cron job. +2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron + unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The + flag protects the *scheduler's* bookkeeping, not the running agent's lease. +3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update + machinery active minutes earlier (`.update-/plugin` capability checks at 18:05:41 and 18:19:53, + `plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the + in-flight job. + +## Open questions + +- Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap? + What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)? +- The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a + reload deferral weight long cron runs differently — or should the cron lease survive a supervisor + relaunch, as the `restart_safe` flag implies? +- Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at + 19:23 — is the double signal expected? + +## Impact + +- One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions + (day closed **160 `completed` / 4 `failed`**). +- `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in + the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**. +- Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time. + +## Related + +- [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers +- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout +- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit +- [[systems/hermes-agent]] — gateway lifecycle record +- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before diff --git a/index.md b/index.md index 015501c..41909cd 100644 --- a/index.md +++ b/index.md @@ -3,15 +3,15 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-01 +verified_on: 2026-11-01 confidence: high tags: [index, infrastructure, systems] --- # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-31 (nightly audit — CT460's 5 h 30 m host outage is documented (PVE `pve` reboot 2026-10-30, ≈39 cron slots lost, 0 failures, 8 catch-up runs at 16:37:21), and the "auto-paused" Mnemosyne Health Monitor turned out to have been *running*; it closed `completed` at 10:02:48, its incident auto-closed, and the job is enabled again; 2026-10-30 closed 138 / 0) +> model routing and automation. Last updated: 2026-11-01 (nightly audit — Conduit's plugin self-updated to **`0.122.2`** on 10-31 18:05, its uncommitted memory-timeout patch is now **upstream** and the tree is clean, and the call store was re-shaped (**`history` empty**); a **new fault** is documented — the app's "local" voice mode never applies because the Mac mini speech server's **LLM proxy is disabled** (**501**, `llm_proxy.requests: 0`) and its realtime socket rejects `session.update`; the same evening's **18:06:41** gateway reload force-drained after the full 1,800 s cap and orphaned a cron run at 18:37:38, leaving 10-31 at **160 / 4**; both Asana paths on CT460 are still non-functional) > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -43,13 +43,16 @@ tags: [index, infrastructure, systems] - [[systems/tool-search]] — Always-on lazy discovery for external tools ### Integrations -- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.69.2` (HEAD `942ddb8`, 2026-10-24 15:35 — *"Pulse: list candidates, decide in code, journal them (#342)"*; 24 commits on since the `0.64.0` baseline, **0 behind `origin/main`**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE** (the 10-24 "not live" note is superseded). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** Still carries the uncommitted in-tree `dashboard/plugin_api.py` patch. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. +- [[systems/conduit]] — **Conduit: Hermes on Anthony's phone** (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree `~/.hermes/plugins/conduit_push`, **v`0.122.2` (HEAD `9eac857`, 2026-10-31 12:04 — *"Every profile that turns Conduit on runs the main install (#482)"*; **+148 commits** on since the `0.69.2` / `942ddb8` baseline, **working tree clean**)**, upstream `kaishi00/hermes-conduit-notifier`. **Gateway on `:9119` is now pid 176771, started 2026-10-31 19:35:02** (superseded: restarted 2026-10-25 18:42:33, which is when the upstream commits went live). One registered device (iPhone app **2.2.0**). ✅ **The first two real calls in the product's history are now on record — 2026-10-29 08:38:04 and 08:49:36** — after three faults in series were repaired (truncated speech-server URL → `…/v1/realtime`; 12 h dashboard session TTL → 30 d; dead `groq` STT → `local`, dead `openai` TTS → `ollama_local`) → [[incidents/2026-10-29-conduit-voice-silence]]. **So the memory fix is no longer evidence-free.** ✅ **That patch is gone** — the tree is clean and **`MEMORY_TIMEOUT_S` ships upstream** in `dashboard/api/memory.py`. ⚠ **New fault: "local" voice mode never applies** — the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` reads **`llm_proxy.requests: 0`**, realtime socket rejects `session.update`), while the app's "test connection" checks **socket reachability only** → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]. The call store was re-shaped: **`history` is empty** and `last_request_at` is `null`, so the two 10-29 calls are no longer in it. The 10-24 refusal fault has not recurred → [[incidents/2026-10-24-conduit-call-request-refused]]. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26** (peer offline **36 d**, `tx 312 / rx 0`; host answers neither `:8065` nor ICMP). **268** connect timeouts on 10-25 vs 264 on 10-24; the reconnect **attempt counter reset to 54** at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost. - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. - [[systems/scheduled-tasks]] — **As of 2026-10-31 00:3x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 4 → 3; failing 1 → 0; unclosed incident rows 2 → 1; 2026-10-30 closed 138 / 0 after a 5 h 30 m CT460 host outage (PVE reboot, ≈39 slots lost, 8 catch-up runs at 16:37:21), and `ea79a2107e36` is ENABLED again (`last_status: ok`, streak 0) after its 00:01:08 slot closed `completed` at 10:02:48; deliveries 8/8 `delivered`.** **Prior window — as of 2026-10-30 03:3x:** 29 jobs (25 enabled, 4 disabled); non-clean `last_status` 3 → 4; unclosed incident rows 1 → 2; 2026-10-29 settled 177 / 0 (the first clean cron day since 10-24) with 10 deliveries all `delivered`. ⚠ `ea79a2107e36` (Mnemosyne Health Monitor) was AUTO-PAUSED at 01:04:22 after its 00:01:08 slot closed `unknown` on a gateway-connection failure — and its failure notice was NEVER DELIVERED (`deliveries.db` has zero rows for 2026-10-30), so the incident row is the only trace → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. Every other previously-red job has cleared to `ok`/streak 0; the retired DHL row was removed from the job table.** Prior window — as of 2026-10-30 00:0x: 29 jobs (26 enabled, 3 disabled); non-clean `last_status` 5 → 3 (all three are stale re-runs of old faults and every `failure_streak` read 0); unclosed incident rows 4 → 1; 2026-10-29 settled 176 / 0 — the first clean cron day since 10-24, with 10 deliveries all `delivered`. The Job Radar's two-stage split worked end to end for the first time: stage 1 wrote its handoff (170 discovered / 33 duplicates avoided / 37 verified) and stage 2 landed 20 pages + 2 cover letters with `missing_cover_letter_count 0` (Notion 455 → 475). Both stage-1 incident rows closed `resolved` at 10:16:12; the lone survivor is the Facebook monthly's `0e753e_e7f8194a7d06`.** Prior window — as of 2026-10-29 00:3x: 29 jobs, 5 non-clean, 4 unclosed; 2026-10-28 settled 168 / 2 — radar stage 1 hit the iteration cap (no handoff, both stage-2 slots no-oped) and the 03:00 stocktake failed on a 400 that never alerted; 2026-10-27 settled 173 / 1 (revised). Prior window — as of 2026-10-28 00:2x: 29 jobs, 5 non-clean, 2 unclosed; 2026-10-27 settled 172 / 1, and the single failure is the radar's second consecutive tool-budget overrun.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (4 h 2 m 46 s), closed **`failed`**, and delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery finished, **no Notion writes and no cover letters**. Incident row `dca848_e93a538ade6b` opened 11:03:53 and is **still open**. The same day the single radar job was **split into two**: stage 1 `dca8482e4f76` (Discovery, 07:00, `deliver: local`) + new stage 2 `5fc292ff0f3b` (Write and Letters, 11:30/14:30, topic 1030, created 17:34:15, two `completed` test fires that evening). The two incident rows the 2026-10-26 planned restart raised **auto-closed 00:38:32**, and the 2026-10-26 orphan class did not recur. → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[runbooks/job-radar]]. **Prior window — as of 2026-10-27 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, 3 unclosed incident rows; 2026-10-26 settled 169 / 3 — and all three failures are one fault.** `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) was interrupted by a **planned** 14:33:35 `SIGTERM` restart that timed the drain out after 1.35 s, killed after **2 h 34 m** on a **new** string (`lost its durable fire claim ownership`), and raised **two** `cron_incidents` rows; the Mnemosyne Watchdog then fired twice and self-resolved at 15:16:12 → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. **New fleet precedent:** a planned stop can orphan a long-running run just as an unclean exit can — and this time it alerted (the 10-20 orphan did not). **Unclosed incident rows 2 → 3**; one closed: `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean, so **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Non-clean `last_status` 5 → 4:** `c0a9fa314542` and `08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as **`error`, streak 2**. **Pins unchanged.** ⚠ `fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed` — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead thread). ⚠ The radar's 07:00 slot delivered **“RUN INCOMPLETE — tool-call budget exhausted”** and its cover-letter pipeline changed that morning → [[runbooks/job-radar]]. **Prior window — as of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean `last_status`, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0.** ⚠ **Both 10-25 failures are `08eb86f02d37` (Gmail Action Digest), which also fired six times that day — and neither raised an incident row** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. ⚠ **The global default reverted to `custom:omni-101` / `smart-route` on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Nous Portal credentials still invalid — all 8 `auxiliary` slots moved to `no-think/freeapi/auto` at 19:13** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). **Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third *consecutive* clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0.** Zero incident rows opened or closed in-window; pins unchanged. ⚠ **Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls** → [[incidents/2026-10-24-conduit-call-request-refused]]. **Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean `last_status`, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four.** The **2026-10-09 provider timeout storm is now fully closed**: `1b23c042e17d` ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so **all 12 are closed** and **unclosed incident rows fell 4 → 2**. ⚠ **New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer** — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (`CONDUIT_MEMORY_TIMEOUT_S`) + degraded-not-aborted on `TokenError`; dashboard restarted 18:54:43, no abort since. **Patch uncommitted** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]]. The 10-22 stocktake overrun did **not** repeat (10-23 03:07 committed `250c735`). Prior window: **As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean `last_status`, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen.** `dca8482e4f76` (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. `08eb86f02d37` cleared (08:30 slot clean; incident resolved 08:37:21) and **2026-10-20 settled at 169 / 0 + 1 `unknown`**. ⚠ **Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]]. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The **`omni-101` 503 fault did not recur** (back to `monitoring`) and 4 incident rows closed → **9 unclosed**. ⚠ **Two new findings:** the **Gitea config backup has not pushed since 2026-08-08** (75 unpushed commits) while its row reads `ok` → [[incidents/2026-08-08-gitea-config-backup-push-dead]]; and a **gateway unclean exit on 2026-10-20** orphaned a digest run — no delivery row, no incident → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. Also: the **global model default moved to `nous` direct** (`meituan/longcat-2.5-preview:free`) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while **20 of 28 still pin it** → [[systems/current-profile]]. Prior window: **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**302 skills** (69 categories) as of the **2026-10-30** stocktake; regenerated daily by the 03:00 job). The generator now **excludes `_archive/`** — the earlier 346/347 counts included it. Freshness at the 2026-10-30 run: **0 stale / 109 fresh / 3 no-date of 112 pages** (`freshness-20261030.txt`). Note: `SKILL.md` files on disk still exceed the index by one (303 vs 302) — the generator de-dups one nested duplicate, so **302 is the correct unique-skill figure**; see [[systems/scheduled-tasks]]. ⚠ The **2026-10-22** slot ran **97 m 58 s** (vs 16 m 01 s on 10-21), finished `completed` with delivery `suppressed` and **committed nothing**; the 10-23 slot closed that hole. ## Incidents +- [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] — **OPEN — the app's "local" voice mode never applies, and the connection test lies.** The test checks **socket reachability only**, while every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects `session.update` as an unknown event**. `voice.mode` is a **phone-side** selector — **no `mode` key exists** in `config.yaml → voice` or `conduit-settings.json` — so a "mode did not apply" cannot be diagnosed on host config. Found via session `20261031_193709_40b9c2` (19:37:09, 30 msgs / 17 tool calls); `voice.local_live.front_model` was repointed to `mistral/voxtral-small-latest` during it and **nothing was changed on the Mac**. → [[systems/conduit]] +- [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] — **A gateway reload waited its full 1,800 s drain cap, then force-drained and orphaned a live cron run.** At **18:06:41** the gateway took the reload; the drain reported `timed_out=True` at the cap and force-drained at **18:37:38** (new gateway **18:38:43**). `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot, **6 h 38 m 09 s** in) closed **`failed`** at **18:39:15** on *"lost its durable fire claim ownership"*. Same class as [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — **second instance, and the first on a plain reload**. A **second, clean** restart followed at **19:23:39 → 19:24:40**; 2026-10-31 closed **160 / 4**. → [[systems/scheduled-tasks]], [[systems/hermes-agent]] +- [[incidents/2026-10-31-asana-access-failures]] — **Both Asana paths on CT460 are non-functional.** The MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never authenticated** — `~/.hermes/mcp-tokens/` holds only `asana.client.json` (**319 B**, mtime 2026-10-26 01:07) and the logs carry *"MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run `hermes mcp login asana` interactively first"*; the hosted connector also rejects the call shape. The fix needs an **interactive** `hermes mcp login asana`, which a cron context cannot perform. → [[systems/hermes-agent]] - [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] — **RESOLVED (root cause not established).** CT460 went dark for **5 h 30 m on 2026-10-30**: the PVE host `pve` rebooted (boot **16:24:00**), CT460 returned **16:28:00**, the previous gateway life (`pid=3913841`) was logged as exited **UNCLEANLY** at **16:35:50** (*"no exit path ran — SIGKILL / OOM / VM death"*), and the scheduler resumed at **16:37:21** with **8 catch-up runs**. **Zero execution rows exist between 10:58:20 and 16:37:21**; the day still closed **138 / 0** against 10-29's 177 / 0 (≈39 slots lost, no failures). **No monitor alerted** — the two jobs that could have (`cbfe434a3b5e`, `1a8e90ea2271`) are disabled. - [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — **RESOLVED 2026-10-30T10:02:48** (its 00:01:08 slot was still *running*, not dead: it closed `completed` after 10 h 01 m 40 s and delivered, auto-closing `ea79a2_d93c3acdc4f2`; `ea79a2107e36` is enabled again with `last_status: ok` / streak 0). **The Mnemosyne Health Monitor cron job was auto-paused overnight and its failure notice never reached Telegram — the only trace is the incident row.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, `detected`) — and the scheduler **paused the job at that moment**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; live `last_status: error`, `failure_streak: 1`. ⚠ **Two traps in one fault:** the failure notice itself failed to send (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed` — `deliveries.db` has **zero rows for 2026-10-30**), and the run closed `unknown` rather than `failed`. The job was **not re-enabled** and the gateway was **not restarted** by the stocktake — root cause is **unverified**. → [[systems/scheduled-tasks]], [[systems/mnemosyne]] - [[incidents/2026-10-29-conduit-voice-silence]] — **Every Conduit voice call connected and then died silently; three faults sat in series, and the first two real calls in the product's history were placed on 2026-10-29 after all three were fixed.** (A) `voice.speech_server.url` was the truncated `wss://…:8443/v1/` — the plugin dials it **verbatim** and the bare path answers **403**, reported as *"the speech server refused the key."* (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session gets **WS 4401** with no Hermes turn and no error on the device (the phone re-authenticated at 08:00:59, minutes *after* a silent call). (C) Both host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by calls at **08:38:04** and **08:49:36**. ⚠ Two TTL keys exist — `dashboard.basic_auth.session_ttl_seconds` (raised) vs `dashboard.session_ttl_seconds` (**still 43 200**). `status: resolved`. @@ -98,8 +101,8 @@ tags: [index, infrastructure, systems] - [[infrastructure/backups]] — hermes backup CLI, retention, and gaps ### Networking -- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14) -- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **48-node** tailnet inventory (re-verified 2026-10-19: `guest-vps`, `openmausbot`, `openmausbot-vm` added; `mattermost` offline 29d) +- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 9.2.10 + `m93p` 9.2.10, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 76.43 % / ~169 GiB free, 2026-10-31) +- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **50-node** tailnet inventory (re-verified 2026-10-31: `mattermost` back **online** after ~40 days; 7 offline ages refreshed) - [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly ## Concepts diff --git a/log.md b/log.md index 2d17700..0faa291 100644 --- a/log.md +++ b/log.md @@ -1681,3 +1681,55 @@ recorded wrong.** `incidents/2026-10-30-mnemosyne-health-monitor-paused.md`. - **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned `running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written. + +## [2026-11-01] audit | Nightly review — Conduit 0.122.2, the Local Live fault, a force-drained reload, dead Asana paths +- **NEW PAGE — [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].** The app's **"local" voice mode + never applies** even though its connection test succeeds — because the test checks **socket reachability only**. + Every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → + **HTTP 501**, *"start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`** + across 25 observed turns, and its realtime socket **accepts the connection but rejects `session.update` as an + unknown event**. `voice.mode` is a **phone-side** selector: **no `mode` key exists** in `config.yaml → voice` + or `conduit-settings.json` (whose only device state is `devices//mode: small` ×3), so "the change did not + apply" cannot be diagnosed on host config. Evidence from session `20261031_193709_40b9c2` (started **19:37:09**, + 30 msgs / 17 tool calls, still open at the 00:2x reading); `voice.local_live.front_model` was repointed to + `mistral/voxtral-small-latest` mid-session (`config.yaml` mtime **21:02:30**). **Nothing was changed on the + Mac.** ⚠ One misread was caught in-session and is recorded as a lesson: `local_live.py:882` is the **only** + writer of `_local_live_last_request`, inside the chat-completions route — so a "recent `last_request_at`" was + the investigation's **own probe**, not phone traffic (`last_request_at` is `null`). +- **NEW PAGE — [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].** The **18:06:41** gateway reload + waited the **full 1,800 s** drain cap and force-drained at **18:37:38** (new gateway **18:38:43**), orphaning + `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot*, **6 h 38 m 09 s** in) which closed **`failed`** at + **18:39:15** on *"lost its durable fire claim ownership"*. Derivative watchdog reds at 18:39:23 / 18:45:23 / + 19:00:22 cleared **19:15:21**. A **second, clean** restart followed at **19:23:39 → 19:24:40**. Same class as + the 10-26 planned-restart orphan — **second instance, first on a plain reload**. +- **NEW PAGE — [[incidents/2026-10-31-asana-access-failures]].** Both Asana paths on CT460 are non-functional: + the MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never + authenticated** (`~/.hermes/mcp-tokens/` holds only `asana.client.json`, **319 B**, mtime 2026-10-26 01:07; + logs read *"MCP OAuth setup failed for 'asana' … no cached tokens found. Run `hermes mcp login asana` + interactively first"*), and the hosted connector rejects the call shape. The repair needs an **interactive** + login, which cron cannot perform. +- **UPDATED — `systems/conduit.md`** (13,184 → 18,722 B; frontmatter `updated` / `verified_on` → 2026-11-01). + Version `0.69.2` / `942ddb8` → **`0.122.2` / `9eac857`** (10-26 reading kept explicitly superseded); the + ⚠ *"uncommitted patch, five releases behind"* warning is **RESOLVED** — the tree is clean and + **`MEMORY_TIMEOUT_S = 10.0` ships upstream** in `dashboard/api/memory.py` (with a `wait_for` backstop and + `tests/test_memory.py`), so the 10-23 in-tree edit is no longer needed; dashboard gateway + `pid 4042003` (10-29 08:03:26) → **`pid 176771` (10-31 19:35:02)**; the **call-store shape change** is recorded + (`v: 1`; **`history` empty**; **`last_request_at: null`**; `ends` = 2 × `outcome: stopped` at 21:06:05 / + 21:10:48; `settings.quiet_at_screen: true` = #479); data volume 930.0 MB → **938.2 MB** (983,793,664 B) with + **2,623,879 `events` / 4,384 `buckets` / 504 `windows`** (≈8 MB/day, no retention policy); new dated section; + `Open items` — patch item marked resolved, three new local-live items added. +- **UPDATED — `systems/scheduled-tasks.md`** (same pass): fleet **29 jobs (26 enabled / 3 disabled)**; 2026-10-31 + closed **160 / 4**; non-clean `last_status` **4** (2 `error` + 2 `delivery_failed`); failing jobs **1** + (`ea79a2107e36`, `failure_streak` 2); superseded readings marked rather than overwritten. +- **UPDATED — `systems/hermes-agent.md`:** Asana MCP auth-state re-verification note (enabled but never + authenticated) + both 2026-10-31 gateway restarts (18:06:41 force-drain, 19:23:39 → 19:24:40 clean). +- **UPDATED — `index.md`:** frontmatter and banner → **2026-11-01**; Conduit entry (version, patch resolved, + gateway pid, new fault, empty call history); **three new incident entries** added above the 10-30 pair. +- **UPDATED — `log.md`:** this entry. +- **Checked, unchanged:** ship rows before 10-30 (nothing new); `inc_20261031_000001` (gateway-update) is **still + open**, `inc_20261030_000001` closed 2026-10-30 18:05; **3** `deliveries.db` rows for 2026-10-31 carry a + failure error (`child run failed: exit status 1`) — they are the failing runs' own notices, not new faults; + no page archived; **no secrets, tokens, passwords or keys written**. +- **Probes (context, not findings):** `omniroute-1` `100.78.63.20:20128` → **HTTP 200**; `m93p` `100.104.51.98` + and `pve` `100.96.100.82` answered neither ICMP, SSH nor `:8006` from CT460 at the 00:2x reading — + recorded as **unverified reachability from this container**, not as a host outage. diff --git a/systems/conduit.md b/systems/conduit.md index 149201c..461ad4c 100644 --- a/systems/conduit.md +++ b/systems/conduit.md @@ -3,8 +3,8 @@ title: Conduit type: system status: active created: 2026-10-24 -updated: 2026-10-30 -verified_on: 2026-10-30 +updated: 2026-11-01 +verified_on: 2026-11-01 review_after: 2026-11-08 confidence: high tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl] @@ -36,7 +36,8 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste ## Where it lives - **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push` -- **Version (re-verified 2026-10-26):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version). +- - **Version (re-verified 2026-11-01):** **`0.122.2`** (`plugin.yaml`) — the plugin **updated itself on 2026-10-31 18:05** (`plugin.yaml` mtime) and fetched at **18:19** (`FETCH_HEAD`); git HEAD **`9eac857`**, *"Every profile that turns Conduit on runs the main install (#482)"*, upstream commit **2026-10-31 12:04:31 AWST**. **Working tree clean**; **+148 commits** on from the `0.69.2` reading below (`#342` → `#482`). See the 2026-11-01 section at the foot of this page. +- ⚠ **Superseded (2026-10-26 reading):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version). - **⚠ Superseded provenance:** the 2026-10-24 audit read `0.69.0` / `f000bf0`; two further commits landed on 2026-10-24 15:35. Git HEAD was previously recorded as: `- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` — *“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the @@ -44,7 +45,7 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version). - **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push) -- ⚠ **The working tree still carries the uncommitted local patch** (`M dashboard/plugin_api.py`, the +- ✅ **The uncommitted local patch is GONE — and it landed upstream.** `git status --short` on the 0.122.2 tree is **clean**, and `dashboard/api/memory.py` now ships **`MEMORY_TIMEOUT_S = 10.0`** applied to every provider call with an `asyncio.wait_for(…, MEMORY_TIMEOUT_S + 2.0)` backstop, plus its own `tests/test_memory.py`. So the fix in [[incidents/2026-10-22-conduit-memory-start-timeout]] is **upstream, not lost** — the 10-23 in-tree edit is no longer needed. ⚠ **Superseded:** the working tree still carries the uncommitted local patch (`M dashboard/plugin_api.py`, the memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]) — now **five releases behind HEAD** (`0.64.0` → `0.69.0`), so the chance a plugin update resets the tree and drops it is rising, not falling. @@ -58,7 +59,7 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste contents into the wiki**). ## Current state (re-verified 2026-10-30 00:0x AWST) -- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26** +- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 176771, started 2026-10-31 19:35:02** (4 h 50 m at the 2026-11-01 00:2x probe; `GET /` → 302 to auth) — ⚠ superseded: pid 4042003, started 2026-10-29 08:03:26 — the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream commits remain **not** live (the process predates nothing relevant; the tree still only carries the memory-timeout patch). @@ -74,13 +75,13 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste is a **separate key and still 43200**; check both. - **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits `min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false. -- **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2** +- ⚠ **The call store was re-shaped by 0.122.x: `history` is now EMPTY (0 entries) and `last_request_at` is `null`.** The file is `v: 1` with keys `outcomes` / `delivered` / `told` / `watches` / `waiting` / `seeded` / `ends` / `settings`; `ends` holds **2** turn-end records (both `outcome: stopped`, at **21:06:05** and **21:10:48** on 10-31) and `settings` has gained **`quiet_at_screen: true`** (upstream #479 — alert calls wait while you're at a screen). The two 10-29 calls below are **no longer in the store** — do not cite them as live history. ⚠ **Superseded — recent calls, as recorded on 2026-10-30: two, both 2026-10-29.** `conduit-calls.json → history` now reads **2** (08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind it → [[incidents/2026-10-29-conduit-voice-silence]]. -- **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days; +- **Data volume (2026-11-01 00:2x):** `conduit-health.sqlite3` **938.2 MB** (983,793,664 B) holding **2,623,879 `events`**, **4,384 `buckets`**, **504 `windows`** — +8.2 MB and +22,314 events / +279 buckets / +93 windows since the 10-30 reading: growth is steady at **≈8 MB/day** with **no retention policy**. (`conduit-location.sqlite3` 80 KB, mtime 10-31 12:15; `conduit-phone.sqlite3` 296 KB, mtime 10-31 14:54.) The per-kind event split was not re-measured this pass. ⚠ **Superseded (2026-10-30):** **930.0 MB** (+360 MB in 5 days; +8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced, unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`), **4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**. @@ -172,15 +173,55 @@ cat ~/.hermes/conduit-link-status.json ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3 ``` +## 2026-11-01 00:2x AWST — plugin 0.122.2, a re-shaped call store, and the Local Live fault + +One night, three changes. The plugin **self-updated to `0.122.2`** at **18:05** on 2026-10-31 and the +gateway reloaded for it at **18:06:41**; the dashboard gateway came back as **pid 176771** at +**19:35:02**; and a voice conversation opened at **19:37:09** reported that the app's **"local" voice +mode never applies** despite a passing connection test. + +- **The memory-timeout patch is resolved, not lost.** The tree at `0.122.2` is **clean** + (`git status --short` empty) and `dashboard/api/memory.py` carries **`MEMORY_TIMEOUT_S = 10.0`** + with a `wait_for` backstop plus `tests/test_memory.py`. The 2026-10-23 in-tree edit that + [[incidents/2026-10-22-conduit-memory-start-timeout]] depended on is no longer required. +- **The call store changed shape.** `conduit-calls.json` is `v: 1` (`outcomes` / `delivered` / `told` / + `watches` / `waiting` / `seeded` / `ends` / `settings`); **`history` is empty (0)** and + **`last_request_at` is `null`**. `ends` = **2** turn-end records, both `outcome: stopped`, at + **21:06:05** and **21:10:48**. `settings` gained **`quiet_at_screen: true`** (#479). ⚠ The two + 2026-10-29 calls the page used to cite as live history are **no longer in the store**. +- **New fault — local voice mode.** The Mac mini speech server's **LLM proxy is disabled** + (its LLM endpoint answers **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage` + reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects + `session.update` as an unknown event**. The app's **"test connection" only checks socket + reachability**, which is why it reports success. `voice.local_live.front_model` was changed during the + session to `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}` (was + `mistral/ministral-14b-latest`); `config.yaml` mtime **21:02:30**. **Nothing was changed on the Mac.** + → [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]] +- **Misread, corrected mid-session (keep the lesson):** `local_live.py:882` is the **only** writer of + `_local_live_last_request`, inside the chat-completions route — so a "recent `last_request_at`" was + the investigation's own probe, not phone traffic. `last_request_at` proves an HTTP request, never a call. +- **Environment:** the **18:06:41 gateway reload waited the full 1,800 s cap and force-drained at + 18:37:38**, orphaning the Mnemosyne Health Monitor's 12:01:06 slot. + → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]] +- **Data volume:** `conduit-health.sqlite3` **938.2 MB** (983,793,664 B) — **2,623,879 `events`** / + **4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy. +- **Not re-measured:** per-kind event split, health-table freshness, device pairing state. + ## Open items -- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is - now five releases behind HEAD. +- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean + at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`. - **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome, so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was still `null` → [[incidents/2026-10-29-conduit-voice-silence]]. +- **Local voice mode does not work — see [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].** + The Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` shows + **`llm_proxy.requests: 0`**) and its realtime socket **rejects `session.update` as an unknown event**, + while the app's "test connection" checks **socket reachability only**. Nothing changed on the Mac yet. +- `voice.mode` is a **phone-side** selector: neither `config.yaml → voice` nor `conduit-settings.json` + carries a `mode` key, so a "mode change did not apply" cannot be diagnosed on the host config. - Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/` silently 403s every call. -- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded). +- No retention policy on `conduit-health.sqlite3` (**938.2 MB**, ≈8 MB/day, growing unbounded). - The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was raised. → [[systems/mnemosyne]] diff --git a/systems/hermes-agent.md b/systems/hermes-agent.md index 687fe05..6a6819a 100644 --- a/systems/hermes-agent.md +++ b/systems/hermes-agent.md @@ -38,6 +38,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn - **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto` - **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46 - **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01 + ⚠ **Auth state re-verified 2026-11-01: `asana` is enabled but has never been authenticated.** `hermes mcp list` shows it ✓ enabled (`https://mcp.asana.com/v2/mcp`, `auth: oauth`), but `~/.hermes/mcp-tokens/` holds **only** `asana.client.json` (319 B, mtime 2026-10-26 01:07) — no token file — so every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first`. The other four servers are unaffected → [[incidents/2026-10-31-asana-access-failures]] - **Plugins enabled:** 11 — verified 2026-10-01 - **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]]) - See [[model-providers]] for full breakdown. @@ -227,3 +228,28 @@ and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST: → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] - **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission bookkeeping). Never read a counter as an outage duration. + +## 2026-11-01 00:2x AWST — gateway reloads on 10-31 (one force-drained, one clean) and the Asana MCP auth gap + +- **Two gateway starts on 2026-10-31, and the first one killed a live cron run.** + - **18:06:41** — `SIGUSR1 received (systemctl reload / hermes gateway restart)`: *"Restart requested with + 2 active work unit(s); deferring stop() until they finish (cap=1800s)"*. The countdown ran every 30 s + from **1770 s** to **988 s** (18:20:13) against a single `managed` unit (`pid 397`); a **second + `SIGUSR1`** landed at **18:11:23**. The cap expired at **18:37:38** → *"Skipping `.clean_shutdown` + marker — drain timed out with interrupted agents"*. New gateway **18:38:43**. The 12:01:06 slot of + `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`** at **18:39:15** on *"lost its durable + fire claim ownership"* after **6 h 38 m** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. + - **19:23:39 → 19:24:40** — a second restart, **clean**: `drain took 0.00s, timed_out=False`, + *"Previous gateway exited cleanly — skipping session suspension"*. Current gateway **PID 175269**, active + since **19:23:56** (4 h 43 m at the audit probe). Nothing failed after 19:15:21. +- **Two Telegram adapter rebuilds, both self-recovered:** `05:03:42` and `09:12:11` — + `Fatal telegram adapter error (telegram_network_error): Telegram updater.stop() did not finish before the + network-recovery deadline; rebuilding the adapter instead of reusing an Updater whose lifecycle lock may + still be held. (likely CLOSE-WAIT socket)`. +- **Plugin update machinery was active minutes before the reload** (`.update-/plugin` capability checks + 18:05:41 and 18:19:53, `plugin scan passed` 18:20:22) and the 04:56:54 log line advertised updates for + `claude-subscription-directsdk-experimental`, `conduit_push` and `homeassistant`. **Agent version + unchanged** — `hermes-agent` HEAD is still `bf0203a7` (2026-10-22). +- **MCP inventory unchanged at 5 servers, but `asana` has never authenticated** (see the MCP line above and + [[incidents/2026-10-31-asana-access-failures]]): **7 728** `MCP OAuth setup failed for 'asana'` log lines in + the retained window (≥2026-10-21), **1 156** of them on 10-31. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index d50ae2b..0d57f2d 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-31 -verified_on: 2026-10-31 +updated: 2026-11-01 +verified_on: 2026-11-01 review_after: 2026-11-06 confidence: high tags: [cron, scheduled, automation] @@ -14,7 +14,41 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole +- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway + log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full + 1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled** + (`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` / + 4 `failed`** (+ this audit `running`, started 23:59:17) against **138 / 0** on 10-30 and **177 / 0** on + 10-29. **All four failures are one fault:** `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed` + at 18:39:15** after **6 h 38 m 09 s** on `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its + durable fire claim ownership`, plus **three derivative `96eb95ce12f2`** (log-error watchdog) reds at + 18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) → + [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. **Mechanism (new for this class):** a + **`SIGUSR1` reload at 18:06:41** deferred its stop with *"cap=1800s"*, waited the **entire cap** on one + `managed` work unit (`pid 397`) — the cron unit itself was logged **`restart_safe: True`** and + *excluded* from the wait — then force-drained at **18:37:38** (*"drain timed out with interrupted + agents"*). A **second, clean restart** followed at 19:23:39 → 19:24:40 (`drain took 0.00s, + timed_out=False`; *"Previous gateway exited cleanly"*), gateway PID **175269** up since **19:23:56**, + and **nothing failed after 19:15:21**. **Non-clean `last_status` 3 → 4**, **failing (run failed + outright) 0 → 1** — `ea79a2107e36` reads `error` with **`failure_streak: 2`**, the fleet's *only* + non-zero streak; its next slot (2026-11-01 00:00:41) was already `running` at audit time. **Unclosed + incident rows 1 → 3:** `ea79a2_a4661554a3c2` (first seen **2026-10-26** 14:35:59) was + **re-used/refreshed** (`last_seen` 18:39:15) and **new** row `ea79a2_9197df32320b` opened 18:06:44 + (`alerted` 18:06:48) — two rows, one event, same job. **Delivery health recovered decisively: 8 + deliveries on 10-31, 8/8 `delivered`** (five normal + **three failure notices**) against **zero rows on + 10-30** — the notice path that failed silently on 10-30 worked. ⚠ **The stocktake (`1a8e90ea2271`) ran + 03:00:58 → 06:44:31 (3 h 43 m 33 s)** — its longest run since 10-22 — closed `completed` with delivery + `suppressed` and **committed nothing**, leaving **7 modified files + 1 untracked** + (`raw/stocktake/freshness-20261031.txt`); this audit committed them. ⚠ Two **Telegram adapter rebuilds** + (`telegram_network_error`, *"updater.stop() did not finish … likely CLOSE-WAIT socket"*) at **05:03:42** + and **09:12:11**, both self-recovered. ⚠ **Asana is unreachable on both paths from CT460** — the `asana` + MCP server has never authenticated (`~/.hermes/mcp-tokens/` holds only `asana.client.json`) and the + hosted connector rejects `opt_fields` as a non-list (422); **7 728** OAuth-failure log lines in the + retained window, **1 156** of them on 10-31 → + [[incidents/2026-10-31-asana-access-failures]]. **Sessions in window: 17** (14 `cron`, 3 `desktop` — two + Conduit Pulse sessions and a voice-settings background investigation, the latter two **still running** + at audit time). +- **⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled / 3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136 `completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 → @@ -51,11 +85,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30 10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll). -- **Non-clean `last_status`:** **3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded: - the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36` +- **Non-clean `last_status`:** **4 as of 2026-11-01 00:2x AWST** — **2 `error` + 2 `delivery_failed`** + (`ea79a2107e36` `error` / **`failure_streak: 2`** — the 18:06 reload orphan; `0e753e9720fd` `error` / + 0; `fad5a86979d4` and `066bb5191280` `delivery_failed` / 0). ⚠ Superseded: the 2026-10-31 00:3x reading + was **3** — **1 `error` + 2 `delivery_failed`**; before that, the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36` (Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault. -- **Failing (last run failed outright):** **0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x +- **Failing (last run failed outright):** **1 as of 2026-11-01 00:2x AWST** — + `ea79a2107e36` (Mnemosyne Health Monitor), **`failure_streak: 2`**, last run 2026-10-31 12:01:06 → + 18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was **0**; before that, the 2026-10-30 03:3x reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.