audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4

This commit is contained in:
Tony0410
2026-11-01 00:30:42 +08:00
parent 74a5b2d117
commit 0c826f2b5f
8 changed files with 488 additions and 22 deletions
+51 -10
View File
@@ -3,8 +3,8 @@ title: Conduit
type: system
status: active
created: 2026-10-24
updated: 2026-10-30
verified_on: 2026-10-30
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-08
confidence: high
tags: [conduit, plugin, phone, iphone, push, voice, memory, tool-calling, session-ttl]
@@ -36,7 +36,8 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
## Where it lives
- **Plugin tree:** `/home/hermes/.hermes/plugins/conduit_push`
- **Version (re-verified 2026-10-26):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version).
- - **Version (re-verified 2026-11-01):** **`0.122.2`** (`plugin.yaml`) — the plugin **updated itself on 2026-10-31 18:05** (`plugin.yaml` mtime) and fetched at **18:19** (`FETCH_HEAD`); git HEAD **`9eac857`**, *"Every profile that turns Conduit on runs the main install (#482)"*, upstream commit **2026-10-31 12:04:31 AWST**. **Working tree clean**; **+148 commits** on from the `0.69.2` reading below (`#342` → `#482`). See the 2026-11-01 section at the foot of this page.
- ⚠ **Superseded (2026-10-26 reading):** `0.69.2` (plugin.yaml) / `0.69.2` (dashboard manifest), git HEAD `942ddb8` — *Plugin and relay fixes left over from earlier reviews (0.69.2) (#342)*, committed 2026-10-24 15:35:42 +0000. **0 commits behind `origin/main`**; **24 commits on from the `0.64.0` / `eaa14be` baseline** of the 2026-10-24 audit (upstream `#324`–`#342`: Pulse cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway restart, `channels.yaml`, App Store builds following the archived plugin version).
- **⚠ Superseded provenance:** the 2026-10-24 audit read `0.69.0` / `f000bf0`; two further commits landed on
2026-10-24 15:35. Git HEAD was previously recorded as: `- **Version:** `0.69.0` (`plugin.yaml` and `dashboard/manifest.json`), git HEAD `f000bf0` —
*“Pulse: list candidates, decide in code, journal them (#337)”*. **22 commits on from the
@@ -44,7 +45,7 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
cards/decisions/journal, a Live Activity messaging tool, contact cards, a self-update gateway
restart, `channels.yaml`, App Store builds following the archived plugin version).
- **Upstream:** `https://github.com/kaishi00/hermes-conduit-notifier.git` (origin, fetch+push)
- ⚠ **The working tree still carries the uncommitted local patch** (`M dashboard/plugin_api.py`, the
- ✅ **The uncommitted local patch is GONE — and it landed upstream.** `git status --short` on the 0.122.2 tree is **clean**, and `dashboard/api/memory.py` now ships **`MEMORY_TIMEOUT_S = 10.0`** applied to every provider call with an `asyncio.wait_for(…, MEMORY_TIMEOUT_S + 2.0)` backstop, plus its own `tests/test_memory.py`. So the fix in [[incidents/2026-10-22-conduit-memory-start-timeout]] is **upstream, not lost** — the 10-23 in-tree edit is no longer needed. ⚠ **Superseded:** the working tree still carries the uncommitted local patch (`M dashboard/plugin_api.py`, the
memory-timeout fix in [[incidents/2026-10-22-conduit-memory-start-timeout]]) — now **five releases
behind HEAD** (`0.64.0` → `0.69.0`), so the chance a plugin update resets the tree and drops it is
rising, not falling.
@@ -58,7 +59,7 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
contents into the wiki**).
## Current state (re-verified 2026-10-30 00:0x AWST)
- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 4042003, started 2026-10-29 08:03:26**
- **Dashboard gateway:** listening on `0.0.0.0:9119`; **pid 176771, started 2026-10-31 19:35:02** (4 h 50 m at the 2026-11-01 00:2x probe; `GET /` → 302 to auth) — ⚠ superseded: pid 4042003, started 2026-10-29 08:03:26
— the restart that put the 30-day session TTL live (see below). The `0.64.0` → `0.69.0` upstream
commits remain **not** live (the process predates nothing relevant; the tree still only carries the
memory-timeout patch).
@@ -74,13 +75,13 @@ Related: [[systems/mnemosyne]] (memory backend the voice session reads), [[syste
is a **separate key and still 43200**; check both.
- **Call settings:** `enabled`, `when_asked`, `decides`, `alerts` all true; rate limits
`min_gap_s 120` / `per_hour 6` / `per_day 20`; `avoid_unknown_place` false.
- **Recent calls — two, both 2026-10-29.** `conduit-calls.json → history` now reads **2**
- ⚠ **The call store was re-shaped by 0.122.x: `history` is now EMPTY (0 entries) and `last_request_at` is `null`.** The file is `v: 1` with keys `outcomes` / `delivered` / `told` / `watches` / `waiting` / `seeded` / `ends` / `settings`; `ends` holds **2** turn-end records (both `outcome: stopped`, at **21:06:05** and **21:10:48** on 10-31) and `settings` has gained **`quiet_at_screen: true`** (upstream #479 — alert calls wait while you're at a screen). The two 10-29 calls below are **no longer in the store** — do not cite them as live history. ⚠ **Superseded — recent calls, as recorded on 2026-10-30: two, both 2026-10-29.** `conduit-calls.json → history` now reads **2**
(08:38:04, 08:49:36). The 08:38:04 call produced no Mac connection and no Hermes turn, and the
iPhone re-authenticated at 08:39:05 — the expired-session fault. The 08:49:36 call was placed after
the `classic` mode switch and the STT/TTS fixes, but the session ended one second earlier, so its
outcome is **unrecorded**. The memory-timeout patch therefore still has no *confirmed* call behind
it → [[incidents/2026-10-29-conduit-voice-silence]].
- **Data volume (2026-10-30 00:0x):** `conduit-health.sqlite3` **930.0 MB** (+360 MB in 5 days;
- **Data volume (2026-11-01 00:2x):** `conduit-health.sqlite3` **938.2 MB** (983,793,664 B) holding **2,623,879 `events`**, **4,384 `buckets`**, **504 `windows`** — +8.2 MB and +22,314 events / +279 buckets / +93 windows since the 10-30 reading: growth is steady at **≈8 MB/day** with **no retention policy**. (`conduit-location.sqlite3` 80 KB, mtime 10-31 12:15; `conduit-phone.sqlite3` 296 KB, mtime 10-31 14:54.) The per-kind event split was not re-measured this pass. ⚠ **Superseded (2026-10-30):** **930.0 MB** (+360 MB in 5 days;
+8.2 MB/25.7 h as measured on 10-25 — the rate has not held, so treat growth as **ingestion-paced,
unbounded**), holding **2,601,565 `events`** (2,592,189 `sample` / 6,146 `sleep` / 3,230 `workout`),
**4,105 `buckets`** and **411 `windows`**; newest health row **2026-10-29 23:12:01 AWST**.
@@ -172,15 +173,55 @@ cat ~/.hermes/conduit-link-status.json
ls -la ~/.hermes/conduit-*.json ~/.hermes/conduit-*.sqlite3
```
## 2026-11-01 00:2x AWST — plugin 0.122.2, a re-shaped call store, and the Local Live fault
One night, three changes. The plugin **self-updated to `0.122.2`** at **18:05** on 2026-10-31 and the
gateway reloaded for it at **18:06:41**; the dashboard gateway came back as **pid 176771** at
**19:35:02**; and a voice conversation opened at **19:37:09** reported that the app's **"local" voice
mode never applies** despite a passing connection test.
- **The memory-timeout patch is resolved, not lost.** The tree at `0.122.2` is **clean**
(`git status --short` empty) and `dashboard/api/memory.py` carries **`MEMORY_TIMEOUT_S = 10.0`**
with a `wait_for` backstop plus `tests/test_memory.py`. The 2026-10-23 in-tree edit that
[[incidents/2026-10-22-conduit-memory-start-timeout]] depended on is no longer required.
- **The call store changed shape.** `conduit-calls.json` is `v: 1` (`outcomes` / `delivered` / `told` /
`watches` / `waiting` / `seeded` / `ends` / `settings`); **`history` is empty (0)** and
**`last_request_at` is `null`**. `ends` = **2** turn-end records, both `outcome: stopped`, at
**21:06:05** and **21:10:48**. `settings` gained **`quiet_at_screen: true`** (#479). ⚠ The two
2026-10-29 calls the page used to cite as live history are **no longer in the store**.
- **New fault — local voice mode.** The Mac mini speech server's **LLM proxy is disabled**
(its LLM endpoint answers **HTTP 501** *"Start the server with `--enable_llm_proxy`"*), `/v1/usage`
reads **`llm_proxy.requests: 0`**, and its realtime socket **accepts the connection but rejects
`session.update` as an unknown event**. The app's **"test connection" only checks socket
reachability**, which is why it reports success. `voice.local_live.front_model` was changed during the
session to `{base_url: http://100.78.63.20:20128/v1, model: mistral/voxtral-small-latest}` (was
`mistral/ministral-14b-latest`); `config.yaml` mtime **21:02:30**. **Nothing was changed on the Mac.**
→ [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]]
- **Misread, corrected mid-session (keep the lesson):** `local_live.py:882` is the **only** writer of
`_local_live_last_request`, inside the chat-completions route — so a "recent `last_request_at`" was
the investigation's own probe, not phone traffic. `last_request_at` proves an HTTP request, never a call.
- **Environment:** the **18:06:41 gateway reload waited the full 1,800 s cap and force-drained at
18:37:38**, orphaning the Mnemosyne Health Monitor's 12:01:06 slot.
→ [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]
- **Data volume:** `conduit-health.sqlite3` **938.2 MB** (983,793,664 B) — **2,623,879 `events`** /
**4,384 `buckets`** / **504 `windows`**; ≈**8 MB/day**, no retention policy.
- **Not re-measured:** per-kind event split, health-table freshness, device pairing state.
## Open items
- Commit or upstream the `plugin_api.py` timeout patch so a plugin update cannot revert it — it is
now five releases behind HEAD.
- ✅ **Resolved 2026-11-01** — the `plugin_api.py` timeout patch is no longer needed: the tree is clean
at `0.122.2` and `MEMORY_TIMEOUT_S` ships upstream in `dashboard/api/memory.py`.
- **Verify a call end-to-end from the phone.** The 2026-10-29 08:49:36 call has no recorded outcome,
so `MEMORY_TIMEOUT_S = 60` is still unexercised by a *confirmed* call, and `last_request_at` was
still `null` → [[incidents/2026-10-29-conduit-voice-silence]].
- **Local voice mode does not work — see [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].**
The Mac mini speech server's **LLM proxy is disabled** (LLM endpoint → **501**, `/v1/usage` shows
**`llm_proxy.requests: 0`**) and its realtime socket **rejects `session.update` as an unknown event**,
while the app's "test connection" checks **socket reachability only**. Nothing changed on the Mac yet.
- `voice.mode` is a **phone-side** selector: neither `config.yaml → voice` nor `conduit-settings.json`
carries a `mode` key, so a "mode change did not apply" cannot be diagnosed on the host config.
- Re-check `voice.speech_server.url` after any Conduit/app update or settings sync — a bare `/v1/`
silently 403s every call.
- No retention policy on `conduit-health.sqlite3` (**930 MB** and growing unbounded).
- No retention policy on `conduit-health.sqlite3` (**938.2 MB**, ≈8 MB/day, growing unbounded).
- The Mnemosyne cold start that motivated the patch is still slow (10–45 s); only the *budget* was
raised. → [[systems/mnemosyne]]
+26
View File
@@ -38,6 +38,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
- **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46
- **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01
⚠ **Auth state re-verified 2026-11-01: `asana` is enabled but has never been authenticated.** `hermes mcp list` shows it ✓ enabled (`https://mcp.asana.com/v2/mcp`, `auth: oauth`), but `~/.hermes/mcp-tokens/` holds **only** `asana.client.json` (319 B, mtime 2026-10-26 01:07) — no token file — so every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first`. The other four servers are unaffected → [[incidents/2026-10-31-asana-access-failures]]
- **Plugins enabled:** 11 — verified 2026-10-01
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
- See [[model-providers]] for full breakdown.
@@ -227,3 +228,28 @@ and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
bookkeeping). Never read a counter as an outage duration.
## 2026-11-01 00:2x AWST — gateway reloads on 10-31 (one force-drained, one clean) and the Asana MCP auth gap
- **Two gateway starts on 2026-10-31, and the first one killed a live cron run.**
- **18:06:41** — `SIGUSR1 received (systemctl reload / hermes gateway restart)`: *"Restart requested with
2 active work unit(s); deferring stop() until they finish (cap=1800s)"*. The countdown ran every 30 s
from **1770 s** to **988 s** (18:20:13) against a single `managed` unit (`pid 397`); a **second
`SIGUSR1`** landed at **18:11:23**. The cap expired at **18:37:38** → *"Skipping `.clean_shutdown`
marker — drain timed out with interrupted agents"*. New gateway **18:38:43**. The 12:01:06 slot of
`ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`** at **18:39:15** on *"lost its durable
fire claim ownership"* after **6 h 38 m** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
- **19:23:39 → 19:24:40** — a second restart, **clean**: `drain took 0.00s, timed_out=False`,
*"Previous gateway exited cleanly — skipping session suspension"*. Current gateway **PID 175269**, active
since **19:23:56** (4 h 43 m at the audit probe). Nothing failed after 19:15:21.
- **Two Telegram adapter rebuilds, both self-recovered:** `05:03:42` and `09:12:11` —
`Fatal telegram adapter error (telegram_network_error): Telegram updater.stop() did not finish before the
network-recovery deadline; rebuilding the adapter instead of reusing an Updater whose lifecycle lock may
still be held. (likely CLOSE-WAIT socket)`.
- **Plugin update machinery was active minutes before the reload** (`.update-<id>/plugin` capability checks
18:05:41 and 18:19:53, `plugin scan passed` 18:20:22) and the 04:56:54 log line advertised updates for
`claude-subscription-directsdk-experimental`, `conduit_push` and `homeassistant`. **Agent version
unchanged** — `hermes-agent` HEAD is still `bf0203a7` (2026-10-22).
- **MCP inventory unchanged at 5 servers, but `asana` has never authenticated** (see the MCP line above and
[[incidents/2026-10-31-asana-access-failures]]): **7 728** `MCP OAuth setup failed for 'asana'` log lines in
the retained window (≥2026-10-21), **1 156** of them on 10-31.
+44 -6
View File
@@ -3,8 +3,8 @@ title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-31
verified_on: 2026-10-31
updated: 2026-11-01
verified_on: 2026-11-01
review_after: 2026-11-06
confidence: high
tags: [cron, scheduled, automation]
@@ -14,7 +14,41 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
# Scheduled Tasks
## Count
- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
4 `failed`** (+ this audit `running`, started 23:59:17) against **138 / 0** on 10-30 and **177 / 0** on
10-29. **All four failures are one fault:** `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`
at 18:39:15** after **6 h 38 m 09 s** on `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
durable fire claim ownership`, plus **three derivative `96eb95ce12f2`** (log-error watchdog) reds at
18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) →
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. **Mechanism (new for this class):** a
**`SIGUSR1` reload at 18:06:41** deferred its stop with *"cap=1800s"*, waited the **entire cap** on one
`managed` work unit (`pid 397`) — the cron unit itself was logged **`restart_safe: True`** and
*excluded* from the wait — then force-drained at **18:37:38** (*"drain timed out with interrupted
agents"*). A **second, clean restart** followed at 19:23:39 → 19:24:40 (`drain took 0.00s,
timed_out=False`; *"Previous gateway exited cleanly"*), gateway PID **175269** up since **19:23:56**,
and **nothing failed after 19:15:21**. **Non-clean `last_status` 3 → 4**, **failing (run failed
outright) 0 → 1** — `ea79a2107e36` reads `error` with **`failure_streak: 2`**, the fleet's *only*
non-zero streak; its next slot (2026-11-01 00:00:41) was already `running` at audit time. **Unclosed
incident rows 1 → 3:** `ea79a2_a4661554a3c2` (first seen **2026-10-26** 14:35:59) was
**re-used/refreshed** (`last_seen` 18:39:15) and **new** row `ea79a2_9197df32320b` opened 18:06:44
(`alerted` 18:06:48) — two rows, one event, same job. **Delivery health recovered decisively: 8
deliveries on 10-31, 8/8 `delivered`** (five normal + **three failure notices**) against **zero rows on
10-30** — the notice path that failed silently on 10-30 worked. ⚠ **The stocktake (`1a8e90ea2271`) ran
03:00:58 → 06:44:31 (3 h 43 m 33 s)** — its longest run since 10-22 — closed `completed` with delivery
`suppressed` and **committed nothing**, leaving **7 modified files + 1 untracked**
(`raw/stocktake/freshness-20261031.txt`); this audit committed them. ⚠ Two **Telegram adapter rebuilds**
(`telegram_network_error`, *"updater.stop() did not finish … likely CLOSE-WAIT socket"*) at **05:03:42**
and **09:12:11**, both self-recovered. ⚠ **Asana is unreachable on both paths from CT460** — the `asana`
MCP server has never authenticated (`~/.hermes/mcp-tokens/` holds only `asana.client.json`) and the
hosted connector rejects `opt_fields` as a non-list (422); **7 728** OAuth-failure log lines in the
retained window, **1 156** of them on 10-31 →
[[incidents/2026-10-31-asana-access-failures]]. **Sessions in window: 17** (14 `cron`, 3 `desktop` — two
Conduit Pulse sessions and a voice-settings background investigation, the latter two **still running**
at audit time).
- **⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136
`completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 →
@@ -51,11 +85,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:** **3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded:
the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
- **Non-clean `last_status`:** **4 as of 2026-11-01 00:2x AWST** — **2 `error` + 2 `delivery_failed`**
(`ea79a2107e36` `error` / **`failure_streak: 2`** — the 18:06 reload orphan; `0e753e9720fd` `error` /
0; `fad5a86979d4` and `066bb5191280` `delivery_failed` / 0). ⚠ Superseded: the 2026-10-31 00:3x reading
was **3** — **1 `error` + 2 `delivery_failed`**; before that, the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):** **0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x
- **Failing (last run failed outright):** **1 as of 2026-11-01 00:2x AWST** —
`ea79a2107e36` (Mnemosyne Health Monitor), **`failure_streak: 2`**, last run 2026-10-31 12:01:06 →
18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was **0**; before that, the 2026-10-30 03:3x
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.