Files
agent-estate-wiki/systems/scheduled-tasks.md
T

1158 lines
157 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-11-07
verified_on: 2026-11-07
review_after: 2026-11-08
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
---
# Scheduled Tasks
## Count
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Current as of 2026-11-05 (window 2026-11-04; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-05 00:5x AWST):** **29 jobs (26 enabled / 3 disabled)**. **11-04 settled: 170 rows - 167 `completed` / 2 `failed` / 1 `running`**; deliveries **8 rows / 8 `delivered`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3** (all `state = alerted`). The two failures are one signature: `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**, streak **2**) and `85edf6772bca` (**Tech & AI Newsletter Digest (Analytical)**, streak **1**) both died on `RuntimeError: Upstream stream failed before completion.` **36 s apart** (09:41:44 / 09:42:20) after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`. The third non-clean row is `066bb5191280` (**Uber Eats Monthly Spend Report**) `delivery_failed`, streak 0. Also in-window: `1a8e90ea2271` (**wiki-stocktake-maintenance**) overran **5 h 41 m 46 s** and reported incomplete (**third consecutive day**), and CT299 was taken offline by a **second** wedged `vzdump` -> [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]; the 07:00 pair -> [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]]. **Corrected 2026-11-05 01:3x AWST (live):** this bullet's first draft read the fleet as **158 jobs / 144 enabled / 14 disabled** and 11-04 as ending `1 item_suppressed`. Three independent reads (`jobs.json`, `hermes cron list`, and the in-flight job count) all give **29 jobs / 26 enabled / 3 disabled**, and **no `item_suppressed` row exists on 11-04** - the 170th row is *this audit*, still `running`. Both errors are corrected across `scheduled-tasks`, `current-state`, `log` and `index`. **11-03 has since settled to 171 `completed` / 1 `failed`** (172 rows; its `running` row completed), making 11-03 and 11-04 the only two consecutive days with a `failed` row since 10-31. ⚠ **`deepseek-paid-smart` cannot currently dispatch on the `omniroute` (CT600) endpoint:** keyed probe 2026-11-05 01:33:12 AWST - `POST http://omniroute:20128/v1/chat/completions` -> **HTTP 503 `ALL_TARGETS_SKIPPED`** (`poolSize 5`, `attempted 0`), while `POST http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, resolving the alias to **`deepseek/deepseek-v4-flash`**. The fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101` = the `omniroute-1` node) - which is why it still completes. An unkeyed `/v1/models` probe returning **401** on both hosts is auth-gating, **not** health. ⚠ **Re-verified
2026-11-06 (stocktake): on `omniroute-1` (`100.78.63.20`) the raw model id `deepseek/deepseek-v4-flash`
returns **HTTP 401 `No active credentials for provider: deepseek`** while the fleet's **`deepseek-paid-smart`
alias returns HTTP 200** and resolves to `deepseek/deepseek-v4-flash`. That 401 is an **alias-routing
artefact, not an outage** — always probe the alias the fleet actually pins, never the raw model id.
→ [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -> [[runbooks/job-radar]] `CL_MODEL` caveat.
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`**
after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26
enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1)
joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`).
**Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**,
13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271`
**03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and
again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt`
untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`).
- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
audit committed both on its behalf). See **Known Issues → Open** below.
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
against **161 / 4** on 10-31, **138 / 0** on 10-30 and **177 / 0** on 10-29 — the first **non-clean**
day in four. **The two `unknown` rows are the whole story of the day:** the Sunday 18:00 **Weekly
Review never ran and never delivered** (*"Gateway connection failed; no local fallback"*, incident
`79f9bb_9c443b602eec` **still `alerted`** at audit time, job reads `error` / `failure_streak: 1`),
and one `96eb95ce12f2` watchdog slot. **Three other jobs were `SIGTERM`ed (`-15`) mid-adoption without
writing a durable terminal state** (`85edf6772bca` @07:30:46, `96eb95ce12f2` and `0e753e9720fd`
@09:02:18) — with **no gateway restart or reload anywhere in the window** (PID `175269` up since
10-31 19:24:40; the relay counter ran unbroken through both signals) →
[[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] and
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. **Delivery health:** 8
delivery rows on 11-01, **7 `delivered` / 1 `failed`** — the failure is `066bb5191280`'s
`thread_id 34795` DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded
`failed` anyway). **Non-clean `last_status` 4 → 2`** (`ea79a2107e36`, `79f9bb41ecf5`).
⚠ **`ecc1fc13655a` reads `error` with `failure_streak: 1` and `last_run 2026-11-01T12:01:00` while
owning NO `executions` row for that slot and NO `cron_incidents` row** — a status field with no
artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31
audit left open closed on 11-01 (02:39:33 ×2 for `ea79a2`, **10:44:12** for
`0e753e_e7f8194a7d06` — the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's
transport flapped all day** (dual-stack `api.telegram.org` ConnectTimeouts, IPv4-literal fallbacks,
a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been
absent ~29 h** (`Reconnecting relay` attempt **336** at 11-02 00:10:47) →
[[systems/messaging-integrations]].
- **Prior window (2026-11-01 00:2x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + gateway
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
4 `failed`** (+ this audit `running`, started 23:59:17) against **138 / 0** on 10-30 and **177 / 0** on
10-29. **All four failures are one fault:** `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`
at 18:39:15** after **6 h 38 m 09 s** on `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
durable fire claim ownership`, plus **three derivative `96eb95ce12f2`** (log-error watchdog) reds at
18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) →
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. **Mechanism (new for this class):** a
**`SIGUSR1` reload at 18:06:41** deferred its stop with *"cap=1800s"*, waited the **entire cap** on one
`managed` work unit (`pid 397`) — the cron unit itself was logged **`restart_safe: True`** and
*excluded* from the wait — then force-drained at **18:37:38** (*"drain timed out with interrupted
agents"*). A **second, clean restart** followed at 19:23:39 → 19:24:40 (`drain took 0.00s,
timed_out=False`; *"Previous gateway exited cleanly"*), gateway PID **175269** up since **19:23:56**,
and **nothing failed after 19:15:21**. **Non-clean `last_status` 3 → 4**, **failing (run failed
outright) 0 → 1** — `ea79a2107e36` reads `error` with **`failure_streak: 2`**, the fleet's *only*
non-zero streak; its next slot (2026-11-01 00:00:41) was already `running` at audit time. **Unclosed
incident rows 1 → 3:** `ea79a2_a4661554a3c2` (first seen **2026-10-26** 14:35:59) was
**re-used/refreshed** (`last_seen` 18:39:15) and **new** row `ea79a2_9197df32320b` opened 18:06:44
(`alerted` 18:06:48) — two rows, one event, same job. **Delivery health recovered decisively: 8
deliveries on 10-31, 8/8 `delivered`** (five normal + **three failure notices**) against **zero rows on
10-30** — the notice path that failed silently on 10-30 worked. ⚠ **The stocktake (`1a8e90ea2271`) ran
03:00:58 → 06:44:31 (3 h 43 m 33 s)** — its longest run since 10-22 — closed `completed` with delivery
`suppressed` and **committed nothing**, leaving **7 modified files + 1 untracked**
(`raw/stocktake/freshness-20261031.txt`); this audit committed them. ⚠ Two **Telegram adapter rebuilds**
(`telegram_network_error`, *"updater.stop() did not finish … likely CLOSE-WAIT socket"*) at **05:03:42**
and **09:12:11**, both self-recovered. ⚠ **Asana is unreachable on both paths from CT460** — the `asana`
MCP server has never authenticated (`~/.hermes/mcp-tokens/` holds only `asana.client.json`) and the
hosted connector rejects `opt_fields` as a non-list (422); **7 728** OAuth-failure log lines in the
retained window, **1 156** of them on 10-31 →
[[incidents/2026-10-31-asana-access-failures]]. **Sessions in window: 17** (14 `cron`, 3 `desktop` — two
Conduit Pulse sessions and a voice-settings background investigation, the latter two **still running**
at audit time).
- **⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136
`completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 →
**≈39 slots lost**. **Zero execution rows exist between 10:58:20 and 16:37:21**: PVE host `pve` rebooted (**boot
2026-10-30 16:24:00**, `ssh … uptime -s`), CT460 came back **16:28:00**, the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) was logged as exited **UNCLEANLY — *"no exit path ran — SIGKILL /
OOM / VM death"*** at **16:35:50**, and the scheduler resumed with an **8-run burst at 16:37:21–16:38:22**.
Root cause **not established** (no OOM evidence; that ledger string is a candidate list, not a diagnosis) →
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **`ea79a2107e36` (Mnemosyne Health Monitor) is
ENABLED AGAIN and clean**: its 00:01:08 slot did not die at 01:04:22 — it **closed `completed` at 10:02:48**
after **10 h 01 m 40 s** and **delivered**, auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at
2026-10-30T10:02:48.662+08:00**; live `last_status: ok`, `failure_streak: 0`. Its **12:01 slot was lost to the
outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing (run failed
outright) 1 → 0**, **unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01).
**`deliveries.db` finished 2026-10-30 with 8 rows, all `delivered`** (latest 18:46:04). 2026-10-31 opened clean
(5 runs, all `completed`).
- **⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **29 — 26 enabled, 3 disabled** (re-verified from live `~/.hermes/cron/jobs.json` at 2026-10-31
00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted **25 enabled / 4 disabled**; the 4th
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:** **4 as of 2026-11-01 00:2x AWST** — **2 `error` + 2 `delivery_failed`**
(`ea79a2107e36` `error` / **`failure_streak: 2`** — the 18:06 reload orphan; `0e753e9720fd` `error` /
0; `fad5a86979d4` and `066bb5191280` `delivery_failed` / 0). ⚠ Superseded: the 2026-10-31 00:3x reading
was **3** — **1 `error` + 2 `delivery_failed`**; before that, the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):** **1 as of 2026-11-01 00:2x AWST** —
`ea79a2107e36` (Mnemosyne Health Monitor), **`failure_streak: 2`**, last run 2026-10-31 12:01:06 →
18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was **0**; before that, the 2026-10-30 03:3x
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):** **1** — `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). It carries `failure_streak: 0` because the *run* succeeded; only the delivery failed. ✅ `066bb5191280` (Uber Eats Monthly Spend Report) was **remediated 2026-11-01** — retargeted from the bare `telegram` target to `telegram:1793951355`; the 2026-10 report generated and was delivered on its 2026-11-01 02:00 slot.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only a rolling window of rows — the live floor is now **2026-10-24 09:22:36** (1,001-row cap), so DB-based recounts undercount older streaks and cannot reproduce tallies for days before that floor. Quote the `jobs.json` counter as authoritative and state the floor whenever a DB recount is cited. Live counters as of 2026-10-30 03:3x AWST: **`ea79a2107e36` = 1; every other job = 0.**
- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`).
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12)
- **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive.
- **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart).
## Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 1`" note above is **superseded**; the job cleared on its own 10-20 07:00 slot and has run clean since, including the successful 2026-10-29 split run. |
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "still carries `last_status: error` … now **streak 2**" note above is **superseded**; the job cleared on its Fri 2026-10-23 08:00 slot and has run clean since. |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 2`" note above is **superseded**; the job cleared on its own daily 12:00 slot after 2026-10-19. |
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "reads `error` (streak 1)" note above is **superseded**; the job ran clean on its Mon 2026-10-26 09:00 slot, so the 10-19 outage has cleared. The **revoked-bot-token** finding above stands as a separate, still-unverified delivery-path question — that Monday run reached the model without raising a 503, which does not by itself prove the token works. |
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 08:30 slot after 2026-10-19. |
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. **Live status re-verified 2026-10-31 00:3x: ENABLED, `last_status: ok`, `failure_streak: 0`; incident `ea79a2_d93c3acdc4f2` closed `resolved` 2026-10-30T10:02:48** (⚠ the 2026-10-30 03:3x DISABLED/`unknown` note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** |
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. |
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | camofox-browser-automation, browser-backend-verification | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. |
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
| 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
| DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant |
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
## Disabled Jobs (not removed) — **3 as of 2026-10-31**
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues
### Resolved 2026-11-02
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
### Resolved 2026-10-30
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
slot was still **running** at the 03:3x reading; the `unknown`/`error` verdict came from the admission
reconciliation, not the run. It **closed `completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**,
auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live:
`enabled: true`, `last_status: ok`, `failure_streak: 0`. **Unclosed incident rows 2 → 1**, **non-clean
`last_status` 4 → 3**, **failing 1 → 0**. ⚠ The pause was real to record, but the "job is disabled / notice
never delivered" state described in [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] lasted only until
10:02:48 — `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04), so the
zero-row/never-delivered reading was a 03:3x artefact of the day having just begun.
- **CT460's 5 h 30 m host outage is closed only because the reboot ended it — nothing was repaired.** PVE `pve`
booted **2026-10-30 16:24:00**, CT460 **16:28:00**, scheduler resumed **16:37:21** with 8 catch-up runs
→ [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **≈39 slots lost, 0 failures.** **No monitor
alerted** — the two jobs that could have (`cbfe434a3b5e` NFS mount monitor, `1a8e90ea2271` m93p LXC boot monitor)
are disabled, and this remains the open item.
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
### Resolved 2026-10-12
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].
- **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot).
### Resolved 2026-10-07
- **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
- **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).
### Resolved 2026-10-05
- **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]].
- The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated *as part of that remediation* — the jobs were moved off it instead. ⚠ **Corrected 2026-10-29:** a live probe shows the pin now returns **HTTP 200 on CT600** (`custom:Omniroute:20128`) — the connection has since been re-authenticated there — while `custom:omni-101` (CT101) still returns **HTTP 401**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run.
- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.**
- **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
### Resolved 2026-09-30
- **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]].
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0.
- **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below).
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):**
`dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`**
(not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
both unrecovered.**
✅ **CLEARED in substance 2026-11-07:** the 11-03 ledger row closed `resolved` at **2026-11-05T12:36:17**, and
`dca8482e4f76` has run clean twice since - **5 h 35 m 35 s** on 11-05 and **1 h 28 m 37 s** on 11-06, both
`completed` / `suppressed`. The two unrecovered candidate sets stay lost; nothing in the ledger is red ->
[[systems/scheduled-tasks]].
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
(10-27, 10-28, 11-02) is a pattern, not bad luck.
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
deliverable is a file, `executions.status` is not evidence — check the artefact.**
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
skills-index regenerated).
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
- **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.
- **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
- **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references.
- **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:<topic>`. See [[systems/messaging-integrations]].
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving).
- **It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20):** the identical `HTTP 503 … resource pressure` string took **9 model calls across 8 jobs** again (06:01, then 07:01 → 12:01), and again the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-20 00:03). **Two bursts, four days apart, same node, same shape, no failover** — and nothing was instrumented during either one. **Action for the next occurrence: while jobs are still failing, run a real completion against `custom:omni-101` and capture the node's host CPU / RAM / queue depth — not afterwards.** The ledger again tagged every new row `failure_type: unknown`, so find these by the **error text**, never by `failure_type`. **Also verified 2026-10-20:** all eight affected jobs read `provider: custom:omni-101`, `model: smart-route` in `jobs.json` — one alias, one node, no diversity. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
### Historical
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
## Related
- [[systems/mnemosyne]]
- [[runbooks/job-radar]]
- [[systems/hermes-agent]]
- [[systems/messaging-integrations]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
---
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
[[incidents/2026-10-09-cron-provider-timeout-storm]].
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
03:00 run today decides whether the streak clears.
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
`agent.log`, so it reported the storm rather than suffering it.
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
00:12). The 2026-10-09 run covers both days.
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
mutate cron state**; retiring the job is open item #1 below.
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
See [[incidents/2026-10-09-memory-recall-relevance]].
### Open items (2026-10-09)
1. ~~**Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.~~ **DONE 2026-10-10** — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See [[log.md]] and the 2026-10-10 audit block below.
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
3. ~~**Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).~~ **DONE** — all eight cleared: five on 2026-10-10 and `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41. (A different pair, `1b23c0_c35703f637b8` + `0e753e_e7f8194a7d06`, remains open for unrelated reasons — see `### Open` above.)
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
---
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
**Non-clean `last_status`: 5 jobs, down from 10.**
| Job | ID | `last_status` | streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `error` | 1 |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
(stocktake) is back to `last_status: ok`.
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
---
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
`[SILENT]` (green). → [[systems/mnemosyne]]
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)
**Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed —
and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on
paper had its delivery hop dead the whole time. Detail below.
**Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit
finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit
itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention
floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer
reproducible from this DB).
**Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at
**03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`);
`dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals
now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06`
(`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and
`1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**).
**Job-by-job (verified against the executions log, `delivery_outcome` included):**
- `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**.
Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof.
- `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030
(32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy —
see the Camofox finding — but it produced and delivered.
- `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its
own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the
job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.
- `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed**
(`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00.
- `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window**
(monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed.
**Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the
previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140),
`1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and
pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on
disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step
skipped on the documented path.
**Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 =
99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly
(`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows;
1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working /
1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to
act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures.
**Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the
`telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401
Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed,
and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending
pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the
Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and
its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has
been corrected: two green runs were read as proof the token worked, when a green cron row only means the
*agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value
not recorded here.
**Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of
`camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G,
swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom →
`/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and
completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented
(`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]]
**Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page
archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).
**Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs.
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-14
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
This is the job's **third failure in six days, all three on the opencode path**: 10-08
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
when the LLM turn dies. There is no green-row alternative — the row went red.
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
uncovered.
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
required) and the same on `omniroute:20128` / 100.88.81.19.
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
→ [[systems/mnemosyne]]
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
Count), `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x)
**Count.** As of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`): **28 jobs —
25 enabled / 3 disabled**, unchanged for a sixth consecutive audit. Window **2026-10-23: 169 `completed` /
0 `failed`** (+ this audit `running`), the third zero-failure day in four; **2026-10-22 settled at 173 / 0**.
**The 2026-10-09 timeout storm closed.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
ran clean on its 10-23 slot, closing the storm's last two rows (`1b23c0_c35703f637b8`,
`1b23c0_103cbe04a79b`) at **08:04**. The storm now holds **zero** unclosed rows and the job reads `ok`
again → [[incidents/2026-10-09-cron-provider-timeout-storm]].
**Incident ledger: unclosed 4 → 2** (no rows opened, two closed). Remaining: `c0a9fa_764558b5ae83`
(`alerted` since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and
`0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01 — Facebook Birthday monthly scrape).
**Non-clean `last_status`: 5 → 4** — `c0a9fa314542` (`error`, streak 1, Mon 10-26), `0e753e9720fd`
(`error`, 0, 11-01), `fad5a86979d4` (`delivery_failed`, Mon 10-26), `066bb5191280` (`delivery_failed`, 11-01).
No new failure signature appeared in the window.
**Pins unchanged** — 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6
`custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` /
`deepseek-paid-smart` (this audit).
**Sessions: 14 in the window** — 13 `cron` + **1 `desktop`** (2026-10-23 18:26:12, 13 messages, the Conduit
phone-call session that exposed the memory-timeout fault below).
**New fault found and fixed in-window (not a cron job):** the **Conduit** voice plugin gave the Mnemosyne
provider start a hard-coded **10 s** budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16)
were abandoned by the memory layer and ran without recall. Patched the same evening to a **60 s** default
(`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades instead of aborting; dashboard
restarted 18:54:43 and **no abort line exists after it**. The patch is **uncommitted** in the plugin's
working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
**Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
(`250c735`: freshness 2 stale / 94 fresh, skills-index 306).
**Pages touched:** `systems/conduit.md` (new), `incidents/2026-10-22-conduit-memory-start-timeout.md` (new),
`incidents/2026-10-09-cron-provider-timeout-storm.md` (status → resolved), `systems/mnemosyne.md`,
`current-state.md`, `index.md`, `log.md`, plus frontmatter added to two legacy incident pages (see log).
**Nothing removed or archived:** no job retired, no system obsolete in the window.
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped;
the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
## Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)
**A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0.** Fleet
unchanged at **28 jobs (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident
rows**, pins unchanged. **Nothing opened and nothing closed in the incident ledger in-window.** 14
sessions: 11 `cron` + 3 `desktop`.
**The window's fault is off-cron.** On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to
test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); every reply asserted the
agent has **no calling capability**, with **zero tool calls** in the session — no `tool_search`, no
`skill_view`, no `conduit_call_user`. The tool exists, is registered for every session, is reachable
through lazy discovery, and the plugin ships the `calling-the-user` skill; his call settings
(`enabled` / `when_asked` / `decides` / `alerts`) were permissive and the device was paired. The first
reply came from `kilo-auto/free`, but the mid-session switch to `smart-route` repeated it — **capability
asserted instead of discovered**, not a broken tool →
[[incidents/2026-10-24-conduit-call-request-refused]].
**Conduit re-verified.** Plugin **`0.69.0` / HEAD `f000bf0`** (22 commits on from the `0.64.0` /
`eaa14be` recorded on 10-24) while the gateway process still dates from **2026-10-23 18:54:43**, so the
upstream commits are **not live** and the memory-timeout patch is **uncommitted and now five releases
behind HEAD**. Device phone-status contact last **10-23 19:10:30**; health ingestion ran to
**10-24 09:59:59**; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** — the 10-24
page's 308 `buckets` / 62 `windows` was a **narrower grouping**, not a collapse. ⚠ **Evidence
correction:** `conduit-calls.json → history` is **empty**, and the 10-22 incident's “two calls closed
cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** turn-ends of that night (`ends` records
any turn-end that fires a watch, `MAX_RECENT_ENDS = 64`) — so the memory fix still has **no positive
call evidence** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
**Method note:** daily tallies are bucketed with `substr(started_at,1,10)` — `date(started_at)`
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).
**Pages touched:** `incidents/2026-10-24-conduit-call-request-refused.md` (new), `systems/conduit.md`,
`incidents/2026-10-22-conduit-memory-start-timeout.md` (correction), `systems/scheduled-tasks.md`,
`current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no job retired, no system
obsolete, no decision reversed in the window.
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
health store with no retention; and a **call that has never been placed** since the memory fix.
## Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x)
**Settled: 167 `completed` / 1 `running` / 2 `unknown`** — the first non-clean day after three
consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at **29 jobs, 26 enabled,
3 disabled**. Nothing was added, removed, disabled or repinned in the window.
### The day's three failure modes, none of which is a repeat of 10-31
| # | Fault | Jobs | Durable record | Page |
|---|---|---|---|---|
| 1 | **`SIGTERM` (`-15`) after adoption, no terminal state** — *"whether side effects ran is unknown"* | `85edf6772bca` 07:30:46, `96eb95ce12f2` + `0e753e9720fd` 09:02:18 | 3 `cron_incidents` rows, all `resolved`; one execution left `unknown` | [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] |
| 2 | **Gateway handoff failure** — *"Cron execution unverified … Gateway connection failed; no local fallback"* | `b63ed5817212` (self-healed 68 s) and `79f9bb41ecf5` | 2 rows; **`79f9bb_9c443b602eec` still `alerted`** | [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] |
| 3 | **`last_status: error` with no execution row and no incident row** | `ecc1fc13655a`, `last_run 2026-11-01T12:01:00` | **none** | same page, "Second silent gap" |
Fault 1 is a **fourth variant** of the orphan class (after 10-20 crash, 10-26 drained-`SIGTERM`, 10-31
cap force-drain) and the first with **no gateway restart anywhere near it** — so it cannot be explained
by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row
over a dead job, there is a **red status with no row at all**.
### The 09:00 pair — cold-start cost meets a 30 s ack window
At **09:00:49.708** the spawner logged `Cron external worker for job '96eb95ce12f2' did not acknowledge
within 30s; leaving the durable execution claim untouched`; the worker then spent **09:00:49 → 09:01:02**
doing a full cold boot (`Plugin discovery complete: 83 found, 69 enabled`) before starting its job at
09:01:13 and being signalled at ~09:02:13. **The 30 s window is smaller than this host's cold-start
time.** Not yet formalised as an Open item because the signal source is unidentified — see the incident
page's open questions.
### Delivery health: 8 rows, 7 delivered
`deliveries.db` for 11-01: 02:39:33 ✔, 03:00:37 ✔, **07:18:50 ✘** (`066bb5191280`,
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`),
07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — **and nothing after 11:01:12**, which is how the
missing Weekly Review was detected. The `066bb5191280` case is the **second variant** documented in
[[systems/messaging-integrations]] (thread-less fallback send, run still marked `failed`).
### Incident ledger movement
- **Closed on 11-01:** `ea79a2_a4661554a3c2` + `ea79a2_9197df32320b` (02:39:33 — the Mnemosyne Health
Monitor pair from 10-26 / 10-31) and **`0e753e_e7f8194a7d06`** (10:44:12) — the Facebook birthday
repin row that had been open since **2026-10-01 09:00:55**, the oldest row on the board. That job's
`delivery_outcome: delivered` on the same run.
- **Opened on 11-01:** 5 rows; **4 resolved, 1 open**.
- **Now unclosed: exactly one** — `79f9bb_9c443b602eec` (`state: alerted`, `acked_at: None`).
### Deliberately not done
- No full vault mirror pass (drift counts recorded in `log.md`; a complete pass is bigger than an
audit's mandate).
- No edits to `### Open` beyond the `066bb5191280` note above: the 10-13 GitHub-token and Camofox
containment items were not re-probed this window, so their lines are left as last verified.
- Nothing removed or archived — no job retired, no system obsoleted, no decision reversed.
**Still open (carried):** the single-node no-failover topology behind every model-calling job; the
GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no
retention; **plus new:** the unreconciled admission files
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
signal source.
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
`96eb95ce12f2`'s 15-minute watchdog slots.
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
`delivered`.**
### Both faults of the day are invisible in the ledger
| # | Fault | Job | What the ledger says | Page |
|---|---|---|---|---|
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
`cron_incidents` is not success.
### Delivery health: 9 rows, 9 delivered
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
fleet's **only** non-clean `last_status`.
### Stocktake artefacts committed on the stocktake's behalf
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
by the next slot.
### Method note (correction carried forward)
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
hours after midnight will read every long job mid-flight.
### Pages touched
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
### Still open (carried + new)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
## Re-verified 2026-11-04 (window 2026-11-03, AWST)
**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1
`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1**
(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new
terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session
compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s**
after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at
13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]].
### Count (live, 2026-11-04)
| Measure | 2026-11-03 window | Change vs 11-02 window |
|---|---|---|
| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged |
| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 |
| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** |
| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** |
| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 |
| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged |
| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged |
| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` |
### What changed in the window
- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the
11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered
when it is `failed`. The cap-produced green row is the anomaly, not the red one.
- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no
handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**,
`hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest
states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference
survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on
[[systems/push-notifications]], which is untouched and still `degraded`.
- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost
`error` did not reappear in this window.
### Files touched by this audit
- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`,
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update),
`log.md`.
- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and
`raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again.
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window).
### Still open (2026-11-04 — carries its predecessor forward)
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified
`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the
**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket;
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
## Re-verified 2026-11-06 (window 2026-11-05, AWST; live reads 2026-11-06 00:1x-00:4x)
### The day was a recovery day, not a repeat
- **Both 11-04 casualties ran clean.** `85edf6772bca` (*Tech & AI Newsletter Digest (Analytical)*, streak 1)
completed its 07:00 run in **7 m 01 s** (07:00:42 -> 07:07:43); its incident row closed at **07:07:42**.
`dca8482e4f76` (*Daily Marketing Job Radar - Discovery*, streak 2, dead twice on 11-04) also completed its
07:00 run - but took **5 h 35 m 35 s** (07:00:42 -> 12:36:18), and that duration is what held its two
incident rows open until **12:36:17**. Neither job re-raised `RuntimeError: Upstream stream failed before
completion.`
- **No `unknown`, `timeout` or `error` row exists anywhere in the 11-05 ledger** - the first such day since
11-02.
- **Deliveries: 10 rows, all `delivered`, none flagged `for_failure`** - 07:02:11, 07:07:42, 07:30:39,
08:06:00, 08:35:30, 10:41:20, 12:15:25, 18:45:49, 19:25:00, 23:27:09 AWST.
### Long runs (completed, not failures)
| Job | Slot | Duration | Against its norm |
|---|---|---|---|
| `ea79a2107e36` Mnemosyne Health Monitor | 00:00 | **7 h 01 m 28 s** (00:00:43 -> 07:02:11) | normal 17-31 m |
| `1a8e90ea2271` wiki-stocktake-maintenance | 03:00 | **3 h 01 m 58 s** (03:00:32 -> 06:02:30) | 11-02 7 h 40 m, 11-03 5 h 41 m |
| `dca8482e4f76` Job Radar - Discovery | 07:00 | **5 h 35 m 35 s** | recovered from 2 failures |
| `5fc292ff0f3b` Job Radar - Stage 2 | 14:30 | **8 h 56 m 17 s** (14:30:53 -> 23:27:10) | delivered 23:27:09 |
| `5fc292ff0f3b` Job Radar - Stage 2 | 11:30 | 10 m 33 s | normal |
| `85edf6772bca` Newsletter Digest | 07:00 | 7 m 01 s | normal |
### The 03:00 stocktake overran a third day - and committed this time
`1a8e90ea2271` ran **03:00:32 -> 06:02:30 (3 h 01 m 58 s)**, so the streak now reads
**7 h 40 m -> 5 h 41 m -> 3 h 02 m** (improving, still ~6x norm), and it **did commit** its artefacts
(`2f3fc99`, 05:55) - against the 11-04 state where it "reported incomplete and committed nothing". The vault
repo `/home/hermes/wiki` picked that up at 05:58 (`ee00b5c`). The nightly audit did **not** have to commit the
skills-index on the stocktake's behalf this pass.
### Sessions in the window
**26 sessions had activity touching 2026-11-05** - 21 `cron`, 4 `subagent`, 1 `desktop`. The only long-form
one is the morning **Mnemosyne Health Monitor** run (`local-b433c51e...`, `cron`, 17 messages, 07:02-07:50):
it requested tool approvals and received denials. No configuration change came out of it, and it is
consistent with that job's 7 h 01 m ballooning above.
### Still open (2026-11-06 - carries its predecessor forward)
Everything on the 2026-11-04 list stands unchanged and not re-probed this window (single-node no-failover
topology; the GitHub Trending Radar delivery token; Camofox containment under multi-tab load; the Conduit
health store with no retention; the unreconciled admission files; the unidentified `-15` signal source; the
30 s worker-ack window vs cold-start; whether a stream-silence timeout should retry a file-deliverable job;
`radar-ref/handoff/` retention). **New this window:**
- **Live secrets sit verbatim in Mnemosyne rows and in two cron prompts.** Three `.env` values (`TELEGRAM_BOT_TOKEN`,
`MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) appear in `memories`, `episodic_memory`,
`working_memory`, `gists` and `memoria_facts.context_snippet` plus the FTS mirrors - **17 rows**, 4 + 6 + 7 per
key - and each of the **7 uncompressed rotation snapshots** reproduces the whole DB; all four values also sit
verbatim in the **1.7 GB session store `state.db`**, the likely capture origin. The Camofox key is embedded in
the prompts of `0e753e9720fd` and `fad5a86979d4` (both **enabled** Facebook Birthday jobs) and therefore in
`cron/deliveries.db` - the only `config-backup/` DB carrying any tested value. **Found, not remediated** ->
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **`ea79a2107e36` ballooned twice out of two runs** (7 h 01 m from the 00:00 slot, then still running 12 h
after its 12:00:45 slot) - against a 17-31 m normal. Cause unexplained; the 12:00 run had delivered nothing
at audit time.
- **`066bb5191280`** (Uber Eats Monthly Spend Report) is now the fleet's **only** non-clean `last_status`
(`delivery_failed`, streak 0) - carried, not re-diagnosed this window.
- **`deepseek-paid-smart` on `omniroute` (CT600) - carried, not re-probed this window.** 11-05 read: HTTP
**503** `ALL_TARGETS_SKIPPED` on `omniroute:20128` vs HTTP **200** on `omniroute-1`
(100.78.63.20:20128 -> `deepseek/deepseek-v4-flash`). This audit remains the fleet's only
`deepseek-paid-smart` pin (provider `custom:omni-101`), which is why it still completes.
### Freshness
`raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes): **0 stale (>=90d), 124 fresh, 3
no-date of 127 pages** - the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
`experiments/active/example.md`, `log.md`).
### Files touched by this audit
- **New:** `incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md`,
`raw/stocktake/freshness-20261106.txt`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
`log.md`, `incidents/2026-11-03-job-radar-stream-active-timeout.md`,
`incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` (both -> `status: resolved` with closure
timestamps), and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.
## Re-verified 2026-11-07 (window 2026-11-06, AWST; live reads 2026-11-07 00:3x-00:5x)
### The day
2026-11-06 closed **172 execution rows across 16 distinct jobs: 170 `completed` / 1 `failed` / 1
`running`** (this audit, 23:59:31 - so the running row is not lost work). Deliveries **10 rows / 10
`delivered` / 0 `for_failure`** (00:14:38, 07:06:34, 07:30:36, 08:29:16, 08:35:53, 12:20:23, 13:26:22,
14:42:58, 18:45:37, 19:48:39). Fleet **29 jobs (26 enabled / 3 disabled)** - unchanged for a fifth
consecutive audit; non-clean `last_status` still **1** (`066bb5191280`, `delivery_failed`, streak 0).
Unclosed incident rows **0 -> 0**: no `cron_incidents` row carries `last_seen_at >= 2026-11-06` and none
is outside `closed`/`resolved`, so the window's one `failed` row **raised no row in the ledger and no
alert** (see below).
The day ends a one-day zero-`failed` streak: 11-05 was the first clean ledger day since 11-02, and
11-06 breaks it again - but by a **new mechanism** with no restart and no upstream provider involvement.
### The window's only failure: a concurrent fire discarded the Job Radar's 11:30 slot
Full page: [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. In short:
- Stage 1 (`dca8482e4f76`) ran **07:00:44 -> 08:29:21 (1 h 28 m 37 s)**, `completed`, `suppressed` - its
second consecutive clean run.
- A **`direct`** fire of `5fc292ff0f3b` (Write and Letters) started **08:27:09** (`source: direct`, not the
scheduler's `builtin`) and ran **4 h 59 m 13 s**, completing and delivering at **13:26:22**.
- The scheduler's **own 11:30:32 slot** had started and was running concurrently; at **13:26:38** it logged
*"fire claim ownership lost; interrupting stale run"* and closed **13:26:44 `failed`** with
`Fire claim ownership lost; stale result was discarded.`
- The **14:30:32** slot then ran a normal cycle (**12 m 27 s**, delivered **14:42:59**) and the day landed in
full: `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`, cache
total **494**, 10 today-dated rows, 20 artifacts present and non-empty, *Run status: COMPLETE*.
- **New signature meaning:** every earlier `lost its durable fire claim` case came from a restart/reload
orphaning a live run ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]],
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]). This one had **no restart** - the
claim was lost to another fire of the same job. `executions.db` holds **exactly one** row whose `error`
contains `fire claim`: this one.
### Long runs in-window (>90 m; normal cycles are 12 m - 1 h)
| Job | Slot | Duration | Outcome |
|---|---|---|---|
| `5fc292ff0f3b` (direct fire) | 08:27:09 -> 13:26:22 | 4 h 59 m 13 s | `completed`, delivered 13:26:22 |
| `5fc292ff0f3b` (11:30 slot) | 11:30:32 -> 13:26:44 | 1 h 56 m 12 s | **`failed`** - discarded as stale |
| `ea79a2107e36` (Mnemosyne Health Monitor) | 12:00:43 -> 19:48:39 | 7 h 47 m 56 s | `completed`, delivered 19:48:39 |
| `dca8482e4f76` (Stage 1) | 07:00:44 -> 08:29:21 | 1 h 28 m 37 s | `completed`, `suppressed` |
| `1a8e90ea2271` (stocktake) | 03:00:32 -> 04:11:35 | 1 h 11 m 03 s | `completed`, `suppressed` |
| `5fc292ff0f3b` (14:30 slot) | 14:30:32 -> 14:42:59 | 12 m 27 s | `completed`, delivered |
**The stocktake's overrun is over.** `1a8e90ea2271` ran **1 h 11 m 03 s** - the four-day streak reads
**7 h 40 m -> 5 h 41 m -> 3 h 02 m -> 1 h 11 m** for 11-03 -> 11-06, and it committed normally.
**The previous audit's open question is closed:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot, which
was still in flight 12 h+ at the last audit with no delivery, **completed at 2026-11-06T00:14:39 after
12 h 13 m 54 s and delivered** (00:14:38). Its next 12:00 slot (11-06) then took **7 h 47 m 56 s** and also
delivered. Ballooned but never lost, and no incident row was ever raised for either.
### Sessions
**15 cron sessions started in the 11-06 window (495 messages / 273 tool calls)**, with **16** sessions
carrying activity (one `desktop` session, 146 messages). Busiest by message count:
`wiki-stocktake-maintenance` 157, `Daily Marketing Job Radar - Discovery` 92,
`Enhanced Weekend Adventure Planner` 82, `Daily Marketing Job Radar - Write and Letters` (14:30) 28,
`Mnemosyne Health Monitor` 27.
### Still open
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] - **still OPEN, untouched.** No
rotation, deletion or scrub happened in-window; not re-litigated by this audit.
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - **NEW, OPEN.** The claim-discard
failure mode above. Candidate fixes are listed on the page; none applied.
- **Watching, not open:** the Mnemosyne Health Monitor's 12:00 slot has now ballooned on consecutive days
(12 h 13 m 54 s on 11-05, 7 h 47 m 56 s on 11-06) and still delivers. No incident row; no delivery loss.
### Freshness
`wiki-maintain.sh` (full run: skills-index regenerated -> freshness audit -> Quartz rebuild + serve restart)
completed at **2026-11-07T00:45:32+08:00**, run **after** this audit's writes:
- **Skills index:** `systems/skills-index.md` regenerated - **302 skills / 68 categories** (unchanged; the
wrapper de-dup fix holds).
- **Freshness: `0 stale (>=21d), 125 fresh, 3 no-date of 128 pages`** ->
`raw/stocktake/freshness-20261107.txt`. Nothing crossed the 21-day threshold. The page count is **+3** on
the 2026-11-05 stocktake's 125 (122 fresh / 3 no-date), consistent with the new pages the last two audits
and this one added - this audit contributes
`incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`.
- **Site:** Quartz parsed **128** Markdown files and emitted **769** files (1 m 02 s); deployed to
`public/` and `hermes-wiki-serve.service` restarted. `public/` is generated output - excluded from the
freshness count by design.
### Files touched by this audit
- **New:** `incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md`,
`raw/stocktake/freshness-20261107.txt`.
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `current-state.md`, `index.md`,
`log.md`, `runbooks/job-radar.md`, and the vault mirror (`/home/hermes/wiki`, both conventions).
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
decision reversed.