audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4
This commit is contained in:
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-31
|
||||
verified_on: 2026-10-31
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-06
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,7 +14,41 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
|
||||
- **Current as of 2026-11-01 00:2x AWST (live `jobs.json` + `executions.db` + `cron_incidents` + gateway
|
||||
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
|
||||
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
|
||||
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
|
||||
4 `failed`** (+ this audit `running`, started 23:59:17) against **138 / 0** on 10-30 and **177 / 0** on
|
||||
10-29. **All four failures are one fault:** `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`
|
||||
at 18:39:15** after **6 h 38 m 09 s** on `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
|
||||
durable fire claim ownership`, plus **three derivative `96eb95ce12f2`** (log-error watchdog) reds at
|
||||
18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) →
|
||||
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. **Mechanism (new for this class):** a
|
||||
**`SIGUSR1` reload at 18:06:41** deferred its stop with *"cap=1800s"*, waited the **entire cap** on one
|
||||
`managed` work unit (`pid 397`) — the cron unit itself was logged **`restart_safe: True`** and
|
||||
*excluded* from the wait — then force-drained at **18:37:38** (*"drain timed out with interrupted
|
||||
agents"*). A **second, clean restart** followed at 19:23:39 → 19:24:40 (`drain took 0.00s,
|
||||
timed_out=False`; *"Previous gateway exited cleanly"*), gateway PID **175269** up since **19:23:56**,
|
||||
and **nothing failed after 19:15:21**. **Non-clean `last_status` 3 → 4**, **failing (run failed
|
||||
outright) 0 → 1** — `ea79a2107e36` reads `error` with **`failure_streak: 2`**, the fleet's *only*
|
||||
non-zero streak; its next slot (2026-11-01 00:00:41) was already `running` at audit time. **Unclosed
|
||||
incident rows 1 → 3:** `ea79a2_a4661554a3c2` (first seen **2026-10-26** 14:35:59) was
|
||||
**re-used/refreshed** (`last_seen` 18:39:15) and **new** row `ea79a2_9197df32320b` opened 18:06:44
|
||||
(`alerted` 18:06:48) — two rows, one event, same job. **Delivery health recovered decisively: 8
|
||||
deliveries on 10-31, 8/8 `delivered`** (five normal + **three failure notices**) against **zero rows on
|
||||
10-30** — the notice path that failed silently on 10-30 worked. ⚠ **The stocktake (`1a8e90ea2271`) ran
|
||||
03:00:58 → 06:44:31 (3 h 43 m 33 s)** — its longest run since 10-22 — closed `completed` with delivery
|
||||
`suppressed` and **committed nothing**, leaving **7 modified files + 1 untracked**
|
||||
(`raw/stocktake/freshness-20261031.txt`); this audit committed them. ⚠ Two **Telegram adapter rebuilds**
|
||||
(`telegram_network_error`, *"updater.stop() did not finish … likely CLOSE-WAIT socket"*) at **05:03:42**
|
||||
and **09:12:11**, both self-recovered. ⚠ **Asana is unreachable on both paths from CT460** — the `asana`
|
||||
MCP server has never authenticated (`~/.hermes/mcp-tokens/` holds only `asana.client.json`) and the
|
||||
hosted connector rejects `opt_fields` as a non-list (422); **7 728** OAuth-failure log lines in the
|
||||
retained window, **1 156** of them on 10-31 →
|
||||
[[incidents/2026-10-31-asana-access-failures]]. **Sessions in window: 17** (14 `cron`, 3 `desktop` — two
|
||||
Conduit Pulse sessions and a voice-settings background investigation, the latter two **still running**
|
||||
at audit time).
|
||||
- **⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
|
||||
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
|
||||
3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136
|
||||
`completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 →
|
||||
@@ -51,11 +85,15 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
|
||||
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
|
||||
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
|
||||
- **Non-clean `last_status`:** **3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded:
|
||||
the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
|
||||
- **Non-clean `last_status`:** **4 as of 2026-11-01 00:2x AWST** — **2 `error` + 2 `delivery_failed`**
|
||||
(`ea79a2107e36` `error` / **`failure_streak: 2`** — the 18:06 reload orphan; `0e753e9720fd` `error` /
|
||||
0; `fad5a86979d4` and `066bb5191280` `delivery_failed` / 0). ⚠ Superseded: the 2026-10-31 00:3x reading
|
||||
was **3** — **1 `error` + 2 `delivery_failed`**; before that, the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
|
||||
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
|
||||
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
|
||||
- **Failing (last run failed outright):** **0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x
|
||||
- **Failing (last run failed outright):** **1 as of 2026-11-01 00:2x AWST** —
|
||||
`ea79a2107e36` (Mnemosyne Health Monitor), **`failure_streak: 2`**, last run 2026-10-31 12:01:06 →
|
||||
18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was **0**; before that, the 2026-10-30 03:3x
|
||||
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
|
||||
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
|
||||
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
|
||||
|
||||
Reference in New Issue
Block a user