886 lines
133 KiB
Markdown
886 lines
133 KiB
Markdown
---
|
||
title: Scheduled Tasks
|
||
type: system
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-11-03
|
||
verified_on: 2026-11-03
|
||
review_after: 2026-11-10
|
||
confidence: high
|
||
tags: [cron, scheduled, automation]
|
||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||
---
|
||
|
||
# Scheduled Tasks
|
||
|
||
## Count
|
||
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
|
||
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
|
||
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
|
||
**96** of the completions are `96eb95ce12f2`'s 15-minute watchdog slots) against **167 / 1 / 2** on
|
||
11-01, **160 / 4** on 10-31 and **138 / 0** on 10-30. **Both 11-01 `unknown` rows resolved on 11-02:**
|
||
the Sunday 18:00 Weekly Review reconciled at **01:27:22** (`be6ec11a8e68428084af9ab18fb148db`,
|
||
`completed` / `delivered`, **7 h 26 m 43 s** after it started) and its incident
|
||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691** →
|
||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] (**now `resolved`; the "lost
|
||
review" was never lost**). `ecc1fc13655a`'s 11-01 ghost `error` also cleared — its 11-02 12:00:33 slot
|
||
ran `completed` / `delivered` at 12:29:26. **Unclosed incident rows 1 → 0 — the first zero in the
|
||
period.** **Non-clean `last_status` 2 → 1:** only `066bb5191280` (Uber Eats Monthly Spend Report,
|
||
`delivery_failed`, `failure_streak: 0`, `last_run 2026-11-01T07:18:57`) — the 11-01 DM-thread miss, which
|
||
**did not recur**; it has **no `executions` row on 11-02 at all**. **Delivery health: 9 rows on 11-02,
|
||
9/9 `delivered`** (01:27:21, 07:30:35, 07:42:21, 08:10:18, 08:41:14, 09:07:20, 10:23:15, 12:29:25,
|
||
18:45:37). ⚠ **Two long runs finished `completed` with `error: NULL` and `delivery_outcome:
|
||
suppressed` after ~9 h — and the first raised no incident:** (a) **`dca8482e4f76`** (Job Radar Stage 1)
|
||
**07:00:33 → 16:01:52 = 9 h 01 m 19 s**, hit the agent iteration cap before the handoff, so **both Stage 2
|
||
slots (`92665337…` 11:30, `05a3f00b…` 14:30 safety net) exited `[SILENT]` and nothing reached Notion** —
|
||
the same class that raised incidents on 10-27 and 10-28 **raised none this time** because the terminal
|
||
state flipped to `completed` → [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]];
|
||
(b) **`1a8e90ea2271`** (daily stocktake) **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `suppressed` — it
|
||
**delivered no report** (unlike 11-01's 03:00:37 delivery) and **left its output uncommitted**
|
||
(`systems/skills-index.md` modified, `raw/stocktake/freshness-20261102.txt` untracked; the 2026-11-03
|
||
audit committed both on its behalf). See **Known Issues → Open** below.
|
||
- **Prior window (2026-11-02 00:4x AWST, live `jobs.json` + `executions.db` + `cron_incidents` +
|
||
`deliveries.db` + gateway log) — the fleet ran clean all day, then failed in three different ways
|
||
and missed a delivery.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged). **Window 2026-11-01
|
||
settled 167 `completed` / 1 `running` / 2 `unknown`** (+ this audit `running`, started 23:59:28)
|
||
against **161 / 4** on 10-31, **138 / 0** on 10-30 and **177 / 0** on 10-29 — the first **non-clean**
|
||
day in four. **The two `unknown` rows are the whole story of the day:** the Sunday 18:00 **Weekly
|
||
Review never ran and never delivered** (*"Gateway connection failed; no local fallback"*, incident
|
||
`79f9bb_9c443b602eec` **still `alerted`** at audit time, job reads `error` / `failure_streak: 1`),
|
||
and one `96eb95ce12f2` watchdog slot. **Three other jobs were `SIGTERM`ed (`-15`) mid-adoption without
|
||
writing a durable terminal state** (`85edf6772bca` @07:30:46, `96eb95ce12f2` and `0e753e9720fd`
|
||
@09:02:18) — with **no gateway restart or reload anywhere in the window** (PID `175269` up since
|
||
10-31 19:24:40; the relay counter ran unbroken through both signals) →
|
||
[[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] and
|
||
[[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]]. **Delivery health:** 8
|
||
delivery rows on 11-01, **7 `delivered` / 1 `failed`** — the failure is `066bb5191280`'s
|
||
`thread_id 34795` DM-thread miss (the thread-less fallback still reaches the DM; the run is recorded
|
||
`failed` anyway). **Non-clean `last_status` 4 → 2`** (`ea79a2107e36`, `79f9bb41ecf5`).
|
||
⚠ **`ecc1fc13655a` reads `error` with `failure_streak: 1` and `last_run 2026-11-01T12:01:00` while
|
||
owning NO `executions` row for that slot and NO `cron_incidents` row** — a status field with no
|
||
artefact behind it. **Unclosed incident rows 1 → 1, but a different row:** all three rows the 10-31
|
||
audit left open closed on 11-01 (02:39:33 ×2 for `ea79a2`, **10:44:12** for
|
||
`0e753e_e7f8194a7d06` — the Facebook-repin row open since **2026-10-01**, 31 days). ⚠ **Telegram's
|
||
transport flapped all day** (dual-stack `api.telegram.org` ConnectTimeouts, IPv4-literal fallbacks,
|
||
a disconnect at 20:51:53, polling restarts at 22:33:30 / 23:12:23) and the **relay adapter has been
|
||
absent ~29 h** (`Reconnecting relay` attempt **336** at 11-02 00:10:47) →
|
||
[[systems/messaging-integrations]].
|
||
- **Prior window (2026-11-01 00:2x AWST, live `jobs.json` + `executions.db` + `cron_incidents` + gateway
|
||
log) — the last five hours of 10-31 ran clean, but an 18:06 gateway reload force-drained at its full
|
||
1 800 s cap and killed a live cron run.** Fleet **29 jobs — 26 enabled / 3 disabled**
|
||
(`jobs.json.updated_at 2026-11-01T00:05:42+08:00`). **Window 2026-10-31 settled 160 `completed` /
|
||
4 `failed`** (+ this audit `running`, started 23:59:17) against **138 / 0** on 10-30 and **177 / 0** on
|
||
10-29. **All four failures are one fault:** `ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`
|
||
at 18:39:15** after **6 h 38 m 09 s** on `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
|
||
durable fire claim ownership`, plus **three derivative `96eb95ce12f2`** (log-error watchdog) reds at
|
||
18:39:23 / 18:45:23 / 19:00:22 (watchdog cleared on its 19:15:21 slot) →
|
||
[[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]]. **Mechanism (new for this class):** a
|
||
**`SIGUSR1` reload at 18:06:41** deferred its stop with *"cap=1800s"*, waited the **entire cap** on one
|
||
`managed` work unit (`pid 397`) — the cron unit itself was logged **`restart_safe: True`** and
|
||
*excluded* from the wait — then force-drained at **18:37:38** (*"drain timed out with interrupted
|
||
agents"*). A **second, clean restart** followed at 19:23:39 → 19:24:40 (`drain took 0.00s,
|
||
timed_out=False`; *"Previous gateway exited cleanly"*), gateway PID **175269** up since **19:23:56**,
|
||
and **nothing failed after 19:15:21**. **Non-clean `last_status` 3 → 4**, **failing (run failed
|
||
outright) 0 → 1** — `ea79a2107e36` reads `error` with **`failure_streak: 2`**, the fleet's *only*
|
||
non-zero streak; its next slot (2026-11-01 00:00:41) was already `running` at audit time. **Unclosed
|
||
incident rows 1 → 3:** `ea79a2_a4661554a3c2` (first seen **2026-10-26** 14:35:59) was
|
||
**re-used/refreshed** (`last_seen` 18:39:15) and **new** row `ea79a2_9197df32320b` opened 18:06:44
|
||
(`alerted` 18:06:48) — two rows, one event, same job. **Delivery health recovered decisively: 8
|
||
deliveries on 10-31, 8/8 `delivered`** (five normal + **three failure notices**) against **zero rows on
|
||
10-30** — the notice path that failed silently on 10-30 worked. ⚠ **The stocktake (`1a8e90ea2271`) ran
|
||
03:00:58 → 06:44:31 (3 h 43 m 33 s)** — its longest run since 10-22 — closed `completed` with delivery
|
||
`suppressed` and **committed nothing**, leaving **7 modified files + 1 untracked**
|
||
(`raw/stocktake/freshness-20261031.txt`); this audit committed them. ⚠ Two **Telegram adapter rebuilds**
|
||
(`telegram_network_error`, *"updater.stop() did not finish … likely CLOSE-WAIT socket"*) at **05:03:42**
|
||
and **09:12:11**, both self-recovered. ⚠ **Asana is unreachable on both paths from CT460** — the `asana`
|
||
MCP server has never authenticated (`~/.hermes/mcp-tokens/` holds only `asana.client.json`) and the
|
||
hosted connector rejects `opt_fields` as a non-list (422); **7 728** OAuth-failure log lines in the
|
||
retained window, **1 156** of them on 10-31 →
|
||
[[incidents/2026-10-31-asana-access-failures]]. **Sessions in window: 17** (14 `cron`, 3 `desktop` — two
|
||
Conduit Pulse sessions and a voice-settings background investigation, the latter two **still running**
|
||
at audit time).
|
||
- **⚠ Superseded by the 2026-11-01 entry above — Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
|
||
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
|
||
3 disabled** (`jobs.json.updated_at 2026-10-31T00:29:59+08:00`). **Window 2026-10-30: 138 executions — 136
|
||
`completed`, 0 `failed`, 2 `running`** (the 16:37 catch-up + this audit), against **177 / 177 / 0** on 10-29 →
|
||
**≈39 slots lost**. **Zero execution rows exist between 10:58:20 and 16:37:21**: PVE host `pve` rebooted (**boot
|
||
2026-10-30 16:24:00**, `ssh … uptime -s`), CT460 came back **16:28:00**, the previous gateway life
|
||
(`pid=3913841`, started 2026-10-28T17:21:58Z) was logged as exited **UNCLEANLY — *"no exit path ran — SIGKILL /
|
||
OOM / VM death"*** at **16:35:50**, and the scheduler resumed with an **8-run burst at 16:37:21–16:38:22**.
|
||
Root cause **not established** (no OOM evidence; that ledger string is a candidate list, not a diagnosis) →
|
||
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **`ea79a2107e36` (Mnemosyne Health Monitor) is
|
||
ENABLED AGAIN and clean**: its 00:01:08 slot did not die at 01:04:22 — it **closed `completed` at 10:02:48**
|
||
after **10 h 01 m 40 s** and **delivered**, auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at
|
||
2026-10-30T10:02:48.662+08:00**; live `last_status: ok`, `failure_streak: 0`. Its **12:01 slot was lost to the
|
||
outage** and is the still-`running` 16:37:21 catch-up. **Non-clean `last_status` 4 → 3**, **failing (run failed
|
||
outright) 1 → 0**, **unclosed incident rows 2 → 1** (survivor: `0e753e_e7f8194a7d06`, due 11-01).
|
||
**`deliveries.db` finished 2026-10-30 with 8 rows, all `delivered`** (latest 18:46:04). 2026-10-31 opened clean
|
||
(5 runs, all `completed`).
|
||
- **⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
|
||
- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd` `error` since 2026-10-01, `fad5a86979d4` and `066bb5191280` `delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:** `dca848_e93a538ade6b` and `dca848_0a2f03b5952b` **both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
|
||
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:** `browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
|
||
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** **Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
|
||
- **Prior window — as of 2026-10-27 00:1x AWST *(superseded by the 2026-10-28 entry above)* (live `jobs.json` + `executions.db`) — a *planned* restart interrupted a live job, and the fleet got a loud failure instead of a silent one.** Window 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are a single fault**: `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot) plus two derivative Mnemosyne Watchdog reds → [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]. 2026-10-25 settled **172 / 2**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled** (`d40085da631c` Important-mail monitor, `f77b7e46bd09` Rhino AgentMail Inbox Watcher, `0a7f933d06e2` AgentMail Rhino Inbox Poll). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (GitHub Trending Radar) and `08eb86f02d37` (Gmail Action Digest) both cleared to `ok`, while `ea79a2107e36` joins as **`error`, streak 2**; `0e753e9720fd` (`error`) and `fad5a86979d4` + `066bb5191280` (`delivery_failed`) unchanged. **Unclosed incident rows 2 → 3:** both `ea79a2_*` rows — 14:33:37 *"Cron execution unverified; reconcile `admissions/724a69108b…json`"* and 14:35:59 *"lost its durable fire claim ownership"* — are still `alerted`, alongside `0e753e_e7f8194a7d06` (`detected` since 2026-10-01). **One row closed:** `c0a9fa_764558b5ae83` auto-closed **09:13:00** when the radar's 09:00 slot ran clean — **every row of the 2026-10-16 503 resource-pressure incident is now closed** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart`, 8 unpinned (global default still `custom:omni-101` / `smart-route`). **One gateway start on 2026-10-26** (14:35:18, after a planned `SIGTERM` at 14:33:35; teardown 54.06 s, `.clean_shutdown` marker **skipped**) → [[systems/hermes-agent]]. ⚠ **`fad5a86979d4`'s 03:08 slot completed but delivered `delivery_failed`** — *Thread 34795 not found*, so the digest fell back to the unthreaded chat; the gateway's own shutdown notice hit the same dead thread at 14:33:37. ⚠ **The radar's 07:00 slot closed `completed` at 09:10 but delivered “RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters — and the same morning its cover-letter PDF pipeline moved to Chromium-over-canonical-HTML with model `deepseek-paid-smart` → [[runbooks/job-radar]]. ⚠ **Off-cron, unchanged: Nous Portal credentials still invalid** → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 17** (15 `cron`, 1 `ios`, 1 `telegram`).
|
||
- **Current as of 2026-10-26 00:1x AWST** *(superseded by the 2026-10-27 entry above)* **(live `jobs.json` + `executions.db`)** — **fleet is clean, but one job misbehaved badly and the global model default moved back.** Window 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times that day** → [[incidents/2026-10-25-gmail-digest-runaway-and-refire]]. 2026-10-24 settled **168 / 0** (the 2026-10-25 entry read 167 — its day was still open). **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled**; all 28 IDs still on this page. **Unclosed incident rows still 2** (`0e753e_e7f8194a7d06` `detected` since 2026-10-01; `c0a9fa_764558b5ae83` `alerted` since 2026-10-19) — **nothing opened or closed in-window, so neither 10-25 failure raised an incident row.** Non-clean `last_status` **4 → 5**: `08eb86f02d37` joins as **`delivery_queued`** with an **empty** `last_error` (never resolved to `ok`), alongside `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`) and `c0a9fa314542` (`error`, streak 1). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `deepseek-paid-smart`, 8 unpinned. ⚠ **Global default reverted to OmniRoute on 2026-10-24 19:06** (`custom:omni-101` / `smart-route`), so the 8 unpinned jobs are back on the same upstream as the pinned ones — **the fleet is a single path again** → [[systems/current-profile]]. ⚠ **Three gateway starts on 2026-10-25** (00:32:20, 18:30:04, 19:27:13; gateway pid 2476414 up since 19:26:33) — the five extra digest fires bracket the evening pair → [[systems/hermes-agent]]. ⚠ **Nous Portal credential still invalid** (`Invalid refresh token`) — context summaries unavailable on 10-25 (20 log lines); all 8 `auxiliary` slots were moved to `no-think/freeapi/auto` on `custom:omni-101` at 19:13 → [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]. **Sessions in window: 21** (18 `cron` — 15 `smart-route`, 2 `free-stack`, 1 `deepseek-paid-smart`; 1 `desktop`; 1 `oneshot`; 1 `telegram`).
|
||
- **Current as of 2026-10-25 00:1x AWST (`jobs.json.updated_at 2026-10-25T00:12:26+08:00`) — a third consecutive clean cron day, and an off-cron fault that is not the fleet's.** Window 2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at 170 / 0** (169 at the 10-24 audit). That makes **10-22 → 10-24 three clean days in a row**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Unclosed incident rows still 2** and **nothing opened or closed in-window:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` still 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window: 11 `cron` + 3 `desktop`** — two of the `desktop` ones are the 12:53–12:56 call requests that the agent refused → [[incidents/2026-10-24-conduit-call-request-refused]]. ⚠ **New off-cron fault, open: the agent told Anthony it cannot make phone calls**, three times, with **zero tool calls**, while `conduit_call_user` was one `tool_search` away → same incident page. *(Superseded by the 2026-10-26 entry above.)*
|
||
- **Current as of 2026-10-24 00:2x AWST *(superseded by the 2026-10-25 entry above)* (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`) — a clean day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23: **169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) ran clean on its 10-23 slot, which closed its **last two incident rows** (`1b23c0_c35703f637b8` + `1b23c0_103cbe04a79b`) — **all 12 rows of the 10-09 storm are closed** and that job reads `ok` again → [[incidents/2026-10-09-cron-provider-timeout-storm]]. **Unclosed incident rows 4 → 2:** `c0a9fa_764558b5ae83` (`alerted` since 10-19, next slot Mon 10-26 09:00) and `0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4:** `c0a9fa314542` (`error`, streak 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` and `066bb5191280` (`delivery_failed` — Mon 10-26 / 11-01). **Pins unchanged:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). **14 sessions in the window:** 13 `cron` + **1 `desktop`** (18:26:12 — the Conduit phone-call session) → [[systems/conduit]]. **The stocktake's 10-22 overrun did not repeat:** its 10-23 03:07 slot ran and committed (`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
|
||
- **Current as of 2026-10-23 00:1x AWST *(superseded by the 2026-10-24 entry above)* (`jobs.json.updated_at 2026-10-23T00:07:25+08:00`) — a zero-failure day, five incidents cleared, and a maintenance job that ran six times long and committed nothing.** Window 2026-10-22: **172 `completed` / 0 `failed`** (+ this audit `running`) — zero failures for the second time in three days, with 4 runs already completed by 00:0x on 10-23. **A DB recount by `started_at` and by `claimed_at` puts 2026-10-21 at 168 / 1**, three below the 171 / 1 the 10-22 audit recorded; retention (floor 2026-10-08) does not explain the delta, which is logged here rather than papered over. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Five incidents closed in-window and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (`0e519874d280`, closed 08:01:40), `06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so **unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`: `1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, `c0a9fa_764558b5ae83`, `0e753e_e7f8194a7d06`). **Non-clean `last_status` falls 9 → 5:** `1b23c042e17d` (`error`, streak 2 — **next slot Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1 — Mon 10-26), `0e753e9720fd` (`error`, 0 — 11-01), `fad5a86979d4` (`delivery_failed` — Mon 10-26 03:00), `066bb5191280` (`delivery_failed` — 11-01). All four cleared rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its first slot of 10-22. **Pins:** 11 `custom:omni-101` / `smart-route`, 8 unpinned (`provider`/`model` null → global default), 6 `custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` / `deepseek-paid-smart` (this audit). ⚠ **Stocktake anomaly:** `1a8e90ea2271` ran **03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, finished `completed` with delivery `suppressed` — and **committed nothing**. It left its regenerated `systems/skills-index.md` uncommitted (mtime 10-22 03:00, with a `## .Archive (41)` section reading **347**) and its diagnosed fixes unapplied → [[incidents/2026-08-15-skills-index-drift]]. **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop` (voice-chat-backend research 11:28–11:32; phone-call tests 13:04 / 13:07 / 13:15, models `mimo-v2.5`, `or-inclusionai_ling-2_6-1t`, `minimax/minimax-m3.1-flash`), 2 `acp` (0 messages), 1 `oneshot` memory-capture probe. → [[systems/skills-index]], [[incidents/2026-08-15-skills-index-drift]]
|
||
- **Current as of 2026-10-22 00:1x AWST *(superseded by the 2026-10-23 entry above)* (`jobs.json.updated_at 2026-10-22T00:11:31+08:00`) — the fleet is clean except for one failure, and it is a failure mode this wiki had not seen.** Window 2026-10-21: **171 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`), which died at 07:40:48 after **40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string, which appears in no 2026-10-21 row → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` stays at 9** (7 `error` / 2 `delivery_failed`) but the composition moved: **`08eb86f02d37` cleared** (its 08:30 slot ran clean; incident `08eb86_472c7b609c80` **resolved 2026-10-21 08:37:21**) while **`dca8482e4f76` re-entered** (streak 1). Survivors, with next run: `dca8482e4f76` (1, **10-22 07:00**), `0e519874d280` (2, **10-22 08:00**), `06a0a1c8704f` (1, **10-22 10:00**), `c8d52904e1d6` (1, **10-22 18:00**), `1b23c042e17d` (2, Fri 10-23 08:00), `c0a9fa314542` (1, Mon 10-26 09:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 11-01 02:00). **Incident ledger: 1 opened / 1 closed, unclosed stays 9** (8 `alerted` / 1 `detected`). **Pins:** 17 of the 28 jobs pin `custom:omni-101` (10 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`), **7 carry no pin at all** (`provider`/`model` null → global default), 1 pins `custom:omni` / `smart-route`. **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the gateway-orphaned run; the 10-21 audit's "168 / 0" was a mid-flight snapshot. **12 sessions in the window, all `cron` — no interactive sessions.** ⚠ **Infrastructure changed in-window: Hermes Agent was updated to `0.22.3` at 22:47 → 22:57 with its pre-update snapshot `skipped`** → [[systems/hermes-agent]], [[infrastructure/backups]].
|
||
- **⚠ Current as of 2026-10-21 00:0x AWST *(superseded by the 2026-10-22 entry above)* (`jobs.json.updated_at 2026-10-21T00:07:08+08:00`) — a clean day, one orphaned run, and one long-running silent failure.** Window 2026-10-20: **168 `completed` / 0 `failed`** (plus 1 `unknown` and this audit) — **the first zero-failure day since 2026-10-12**, ending a seven-day run of failures, and **the `omni-101` 503 fault did not recur** (zero rows carry the resource-pressure string; the last such failure remains 2026-10-19). **Zero incident rows opened and 4 closed** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`, `ecc1fc_21a72aa11cc4`), so **unclosed rows fall 13 → 9** (8 `alerted` / 1 `detected`). ⚠ **The exception is a failure this ledger does not record:** `08eb86f02d37` (Gmail Action Digest, 08:30) was claimed 08:31:03 and closed **`unknown`** when the gateway exited uncleanly — *“Scheduler restarted after this execution's owner exited before a durable terminal state”*; no delivery row exists for the slot and no incident row was raised → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); every `failure_streak` ≤ 2. **Fleet unchanged: 28 jobs — 25 enabled / 3 disabled.** ⚠ **Discovery — green row, dead job:** `e27da6c34bbb` (Config Backup, 11:00) reads `ok` while its Gitea push has failed **every night since 2026-08-08** (75 unpushed commits) → [[incidents/2026-08-08-gitea-config-backup-push-dead]]. **Model pins unchanged** — 20 of 28 jobs still pin OmniRoute — but the **8 unpinned jobs moved upstream to `nous` direct** with the global default change, so the fleet now has two upstream paths. **2 interactive sessions in the window** (`talaria-ios` tax work; a `desktop` turn that received the canned non-answer). → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
- **Current as of 2026-10-20 00:09 AWST *(superseded by the 2026-10-21 entry above)* — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||
- **Total jobs:** **29 — 26 enabled, 3 disabled** (re-verified from live `~/.hermes/cron/jobs.json` at 2026-10-31
|
||
00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted **25 enabled / 4 disabled**; the 4th
|
||
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
|
||
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
|
||
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
|
||
- **Non-clean `last_status`:** **4 as of 2026-11-01 00:2x AWST** — **2 `error` + 2 `delivery_failed`**
|
||
(`ea79a2107e36` `error` / **`failure_streak: 2`** — the 18:06 reload orphan; `0e753e9720fd` `error` /
|
||
0; `fad5a86979d4` and `066bb5191280` `delivery_failed` / 0). ⚠ Superseded: the 2026-10-31 00:3x reading
|
||
was **3** — **1 `error` + 2 `delivery_failed`**; before that, the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
|
||
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
|
||
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
|
||
- **Failing (last run failed outright):** **1 as of 2026-11-01 00:2x AWST** —
|
||
`ea79a2107e36` (Mnemosyne Health Monitor), **`failure_streak: 2`**, last run 2026-10-31 12:01:06 →
|
||
18:39:15 (orphaned by the 18:06:41 gateway reload). ⚠ Superseded: the 2026-10-31 00:3x reading was **0**; before that, the 2026-10-30 03:3x
|
||
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
|
||
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
|
||
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
|
||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||
- **Date-bucketing note (added 2026-10-25):** `executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
|
||
- **Failing (delivery only):** **1** — `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). It carries `failure_streak: 0` because the *run* succeeded; only the delivery failed. ✅ `066bb5191280` (Uber Eats Monthly Spend Report) was **remediated 2026-11-01** — retargeted from the bare `telegram` target to `telegram:1793951355`; the 2026-10 report generated and was delivered on its 2026-11-01 02:00 slot.
|
||
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
|
||
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
|
||
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only a rolling window of rows — the live floor is now **2026-10-24 09:22:36** (1,001-row cap), so DB-based recounts undercount older streaks and cannot reproduce tallies for days before that floor. Quote the `jobs.json` counter as authoritative and state the floor whenever a DB recount is cited. Live counters as of 2026-10-30 03:3x AWST: **`ea79a2107e36` = 1; every other job = 0.**
|
||
- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`).
|
||
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
|
||
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
|
||
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
|
||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12)
|
||
- **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive.
|
||
- **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart).
|
||
|
||
## Job Table
|
||
|
||
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|
||
|---|---|---|---|---|---|---|
|
||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
|
||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `0 7 * * 1-5` | `local` (failures → topic 1030) | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00. ✅ **First full split run 2026-10-29:** 07:00:59 → 10:16:12 (3 h 15 m 13 s), `completed`, wrote `handoff/2026-10-29.json` (170 discovered / 33 duplicates avoided / 37 verified); both of its ledger rows closed `resolved` at 10:16:12. **Delivery is now `local`** — stage 2 owns the report ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 1`" note above is **superseded**; the job cleared on its own 10-20 07:00 slot and has run clean since, including the successful 2026-10-29 split run. |
|
||
| Daily Marketing Job Radar — Write and Letters (stage 2) | `5fc292ff0f3b` | `30 11,14 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Created 2026-10-27 17:34** — lands stage 1's handoff (Notion entries + cover letters + audit + report; no discovery). `custom:omni-101` / `smart-route`; first fires 2026-10-27 17:35 and 17:38 both `completed`. ✅ **First full split run 2026-10-29:** fired 10:13:46, consumed the handoff at 10:40:59, closed `completed` 10:44:48 and delivered the same second — **20 pages + 2 cover letters, `missing_cover_letter_count 0`**, Notion board 455 → 475; the 11:30 and 14:30 slots then no-oped correctly. See [[incidents/2026-10-27-job-radar-budget-exhausted]] |
|
||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "still carries `last_status: error` … now **streak 2**" note above is **superseded**; the job cleared on its Fri 2026-10-23 08:00 slot and has run clean since. |
|
||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
|
||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic |
|
||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, `failure_streak: 2`" note above is **superseded**; the job cleared on its own daily 12:00 slot after 2026-10-19. |
|
||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "reads `error` (streak 1)" note above is **superseded**; the job ran clean on its Mon 2026-10-26 09:00 slot, so the 10-19 outage has cleared. The **revoked-bot-token** finding above stands as a separate, still-unverified delivery-path question — that Monday run reached the model without raising a 503, which does not by itself prove the token works. |
|
||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 08:30 slot after 2026-10-19. |
|
||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day |
|
||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. **Live status re-verified 2026-10-31 00:3x: ENABLED, `last_status: ok`, `failure_streak: 0`; incident `ea79a2_d93c3acdc4f2` closed `resolved` 2026-10-30T10:02:48** (⚠ the 2026-10-30 03:3x DISABLED/`unknown` note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** |
|
||
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
|
||
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "`failure_streak: 2`" note above is **superseded**; the job cleared on its Mon/Thu 08:00 slots after 2026-10-19. |
|
||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | ✅ **Live status re-verified 2026-10-30 03:3x:** `last_status: ok`, `failure_streak: 0` — the "Live `last_status: error`, streak 1" note above is **superseded**; the job cleared on its own daily 07:00 slot after 2026-10-19. |
|
||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram:1793951355 | camofox-browser-automation, browser-backend-verification | — | ✅ **Fixed 2026-11-01:** delivery retargeted to `telegram:1793951355` (was a bare `telegram` target that inherited the dead `thread_id 34795` → "configured thread_id 34795 … was not found; delivered without thread_id"). Also repaired the cron prompt, which pointed at **port 9093** (nonexistent) instead of **9377**, and wired in the tested pipeline `scrape-uber-eats-orders.py` → `build-uber-eats-report.py` → `generate-uber-eats-report.py`. The Oct 2026 report was generated and verified against the live Uber DOM → [[reports/uber-eats/index]]. |
|
||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
|
||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text |
|
||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message |
|
||
| 🦝 Hermes Config Backup to Gitea (Daily) ⚠ **push has failed every night since 2026-08-08 — 75 local commits unpushed, job still reads `ok`** → [[incidents/2026-08-08-gitea-config-backup-push-dead]] | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||
|
||
## Removed Jobs
|
||
| Name | Removed | Reason |
|
||
|---|---|---|
|
||
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
||
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
||
| DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant |
|
||
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
|
||
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
||
|
||
## Disabled Jobs (not removed) — **3 as of 2026-10-31**
|
||
| Name | ID | Disabled since | Reason |
|
||
|---|---|---|---|
|
||
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only. **Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
|
||
|
||
## Known Issues
|
||
|
||
### Resolved 2026-11-02
|
||
|
||
- **The "lost" Weekly Review was never lost — it delivered at 01:27:21, 7 h 26 m 43 s after its 18:00 slot.**
|
||
Execution `be6ec11a8e68428084af9ab18fb148db` (job `79f9bb41ecf5`, `scheduled_instant`
|
||
`2026-11-01T10:00:00+00:00`) started **2026-11-01T18:00:39.125776+08:00** and finally closed
|
||
**2026-11-02T01:27:22.000881+08:00** with `status: completed`, `error: NULL`,
|
||
`delivery_outcome: delivered`; the delivery row is timestamped **01:27:21.802246** and incident
|
||
`79f9bb_9c443b602eec` auto-closed at **01:27:21.606691**. **Same execution row, not a re-run** — the
|
||
`unknown` terminal state recorded at 18:05:13 was replaced when the run wrote its real outcome.
|
||
→ [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] is now **`resolved`**.
|
||
**Method lesson for this audit:** the 2026-11-02 00:4x reading of `status: unknown` + `closed_at: NULL`
|
||
was accurate *at the time of reading* and had inverted 43 minutes later. **A nightly audit that runs
|
||
~2 h after midnight must not treat a null terminal state on a long-running job as final** — the
|
||
reconciler's window is shorter than these runs, so `unknown` on a long job is a timing artefact, not a
|
||
verdict. The same false-negative logic applies to `ecc1fc13655a`, whose 11-01 ghost `error` cleared when
|
||
its 11-02 12:00:33 slot closed `completed` / `delivered` at 12:29:26.
|
||
|
||
### Resolved 2026-10-30
|
||
|
||
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
|
||
slot was still **running** at the 03:3x reading; the `unknown`/`error` verdict came from the admission
|
||
reconciliation, not the run. It **closed `completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**,
|
||
auto-closing incident `ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live:
|
||
`enabled: true`, `last_status: ok`, `failure_streak: 0`. **Unclosed incident rows 2 → 1**, **non-clean
|
||
`last_status` 4 → 3**, **failing 1 → 0**. ⚠ The pause was real to record, but the "job is disabled / notice
|
||
never delivered" state described in [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] lasted only until
|
||
10:02:48 — `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04), so the
|
||
zero-row/never-delivered reading was a 03:3x artefact of the day having just begun.
|
||
- **CT460's 5 h 30 m host outage is closed only because the reboot ended it — nothing was repaired.** PVE `pve`
|
||
booted **2026-10-30 16:24:00**, CT460 **16:28:00**, scheduler resumed **16:37:21** with 8 catch-up runs
|
||
→ [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **≈39 slots lost, 0 failures.** **No monitor
|
||
alerted** — the two jobs that could have (`cbfe434a3b5e` NFS mount monitor, `1a8e90ea2271` m93p LXC boot monitor)
|
||
are disabled, and this remains the open item.
|
||
|
||
### Resolved 2026-10-29
|
||
|
||
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
|
||
- **Conduit voice calls — three faults in series, all fixed and verified by two real calls:** the trunk speech-server URL was truncated (`…/v1/` → **403**, blamed on the key), the dashboard session TTL was the **12 h default** (expired sessions answer **WS 4401** with no error on the phone), and **both** host providers were dead (`tts.provider: openai` → Mac `/v1/speech` **502**; `stt.provider: groq` → **403/Cloudflare 1010**). Repaired 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s**, `stt.provider: local`, `tts.provider: ollama_local`; dashboard restarted **08:03:26**. The path was driven end to end, then **calls at 08:38:04 and 08:49:36** — the first entries in `conduit-calls.json → history`. ⚠ Two TTL keys exist and only one was raised → [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]].
|
||
|
||
### Resolved 2026-10-12
|
||
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
|
||
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||
- **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot).
|
||
|
||
### Resolved 2026-10-07
|
||
- **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
|
||
- **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).
|
||
|
||
### Resolved 2026-10-05
|
||
- **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||
- The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated *as part of that remediation* — the jobs were moved off it instead. ⚠ **Corrected 2026-10-29:** a live probe shows the pin now returns **HTTP 200 on CT600** (`custom:Omniroute:20128`) — the connection has since been re-authenticated there — while `custom:omni-101` (CT101) still returns **HTTP 401**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||
- `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run.
|
||
- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.**
|
||
- **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
|
||
|
||
### Resolved 2026-09-30
|
||
- **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]].
|
||
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0.
|
||
- **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below).
|
||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||
|
||
### Open
|
||
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
|
||
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
|
||
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
|
||
iteration cap before the handoff was written**. Both Stage 2 slots declined (`92665337…` 11:30 and the
|
||
`05a3f00b…` 14:30 safety net both exited `[SILENT]`), so **nothing reached Notion**. This is the same
|
||
class that raised `dca848_e93a538ade6b` (10-27) and `dca848_0a2f03b5952b` (10-28) — whose `error`
|
||
columns are literally the runs' INCOMPLETE reports — yet **2026-11-02 produced no `cron_incidents` row
|
||
at all**, because the terminal state flipped from non-clean to `completed`. Three caps in seven days
|
||
(10-27, 10-28, 11-02) is a pattern, not bad luck.
|
||
→ [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]]. **Detection rule: for a job whose
|
||
deliverable is a file, `executions.status` is not evidence — check the artefact.**
|
||
- **`1a8e90ea2271`** (wiki stocktake) ran **03:00:31 → 12:44:00 = 9 h 43 m 29 s**, `completed` /
|
||
`suppressed`, and **delivered no report** (11-01 delivered at 03:00:37; 11-02 has no delivery row near
|
||
03:00). It left its output **uncommitted** — `systems/skills-index.md` modified and
|
||
`raw/stocktake/freshness-20261102.txt` untracked — which the 2026-11-03 audit committed on its behalf.
|
||
The artefacts themselves are sound (freshness report: **4 stale / 113 fresh** at a >21-day threshold;
|
||
skills-index regenerated).
|
||
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
|
||
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
|
||
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
|
||
- **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.
|
||
- **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
||
- **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references.
|
||
- **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:<topic>`. See [[systems/messaging-integrations]].
|
||
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
|
||
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||
|
||
- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving).
|
||
- **It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20):** the identical `HTTP 503 … resource pressure` string took **9 model calls across 8 jobs** again (06:01, then 07:01 → 12:01), and again the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-20 00:03). **Two bursts, four days apart, same node, same shape, no failover** — and nothing was instrumented during either one. **Action for the next occurrence: while jobs are still failing, run a real completion against `custom:omni-101` and capture the node's host CPU / RAM / queue depth — not afterwards.** The ledger again tagged every new row `failure_type: unknown`, so find these by the **error text**, never by `failure_type`. **Also verified 2026-10-20:** all eight affected jobs read `provider: custom:omni-101`, `model: smart-route` in `jobs.json` — one alias, one node, no diversity. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||
|
||
### Historical
|
||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
||
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
|
||
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
|
||
|
||
## Related
|
||
- [[systems/mnemosyne]]
|
||
- [[runbooks/job-radar]]
|
||
- [[systems/hermes-agent]]
|
||
- [[systems/messaging-integrations]]
|
||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
|
||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
|
||
|
||
---
|
||
|
||
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
|
||
|
||
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
|
||
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
|
||
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
|
||
|
||
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
|
||
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
|
||
every run from 18:00 onward completed. Full timeline, job list and incident table:
|
||
[[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||
|
||
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
|
||
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
|
||
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
|
||
03:00 run today decides whether the streak clears.
|
||
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
|
||
`agent.log`, so it reported the storm rather than suffering it.
|
||
|
||
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
|
||
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
|
||
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
|
||
00:12). The 2026-10-09 run covers both days.
|
||
|
||
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
|
||
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
|
||
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
|
||
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
|
||
|
||
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
|
||
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
|
||
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
|
||
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
|
||
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
|
||
mutate cron state**; retiring the job is open item #1 below.
|
||
|
||
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
|
||
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
|
||
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
|
||
See [[incidents/2026-10-09-memory-recall-relevance]].
|
||
|
||
### Open items (2026-10-09)
|
||
|
||
1. ~~**Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.~~ **DONE 2026-10-10** — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See [[log.md]] and the 2026-10-10 audit block below.
|
||
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
|
||
3. ~~**Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).~~ **DONE** — all eight cleared: five on 2026-10-10 and `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41. (A different pair, `1b23c0_c35703f637b8` + `0e753e_e7f8194a7d06`, remains open for unrelated reasons — see `### Open` above.)
|
||
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
|
||
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
|
||
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
|
||
|
||
|
||
|
||
---
|
||
|
||
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
|
||
|
||
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
|
||
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
|
||
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
|
||
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
|
||
|
||
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
|
||
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
|
||
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
|
||
|
||
**Non-clean `last_status`: 5 jobs, down from 10.**
|
||
|
||
| Job | ID | `last_status` | streak |
|
||
|---|---|---|---|
|
||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
|
||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
|
||
| Daily Marketing Job Radar — Discovery (stage 1) | `dca8482e4f76` | `error` | 1 |
|
||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
|
||
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
|
||
|
||
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
|
||
(stocktake) is back to `last_status: ok`.
|
||
|
||
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
|
||
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
|
||
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
|
||
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
|
||
|
||
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
|
||
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
|
||
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
|
||
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
|
||
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
|
||
|
||
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
|
||
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
|
||
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
|
||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||
|
||
---
|
||
|
||
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
|
||
|
||
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
|
||
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
|
||
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
|
||
|
||
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
|
||
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
|
||
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
|
||
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
|
||
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
|
||
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
|
||
|
||
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
|
||
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
|
||
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
|
||
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
|
||
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
|
||
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
|
||
|
||
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
|
||
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
|
||
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
|
||
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
|
||
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
|
||
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
|
||
|
||
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
|
||
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
|
||
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
|
||
|
||
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
|
||
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
|
||
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
|
||
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
|
||
`[SILENT]` (green). → [[systems/mnemosyne]]
|
||
|
||
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
|
||
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
|
||
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
|
||
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
|
||
|
||
## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)
|
||
|
||
**Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed —
|
||
and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on
|
||
paper had its delivery hop dead the whole time. Detail below.
|
||
|
||
**Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit
|
||
finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit
|
||
itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention
|
||
floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer
|
||
reproducible from this DB).
|
||
|
||
**Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at
|
||
**03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`);
|
||
`dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals
|
||
now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06`
|
||
(`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and
|
||
`1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**).
|
||
|
||
**Job-by-job (verified against the executions log, `delivery_outcome` included):**
|
||
- `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**.
|
||
Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof.
|
||
- `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030
|
||
(32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy —
|
||
see the Camofox finding — but it produced and delivered.
|
||
- `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its
|
||
own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the
|
||
job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.
|
||
- `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed**
|
||
(`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00.
|
||
- `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window**
|
||
(monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed.
|
||
|
||
**Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the
|
||
previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140),
|
||
`1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and
|
||
pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on
|
||
disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step
|
||
skipped on the documented path.
|
||
|
||
**Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 =
|
||
99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly
|
||
(`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows;
|
||
1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working /
|
||
1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to
|
||
act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures.
|
||
|
||
**Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the
|
||
`telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401
|
||
Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed,
|
||
and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending
|
||
pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the
|
||
Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and
|
||
its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has
|
||
been corrected: two green runs were read as proof the token worked, when a green cron row only means the
|
||
*agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value
|
||
not recorded here.
|
||
|
||
**Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of
|
||
`camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G,
|
||
swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom →
|
||
`/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and
|
||
completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented
|
||
(`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]]
|
||
|
||
**Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page
|
||
archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).
|
||
|
||
**Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs.
|
||
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
|
||
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
|
||
gateway runtime env outside the 15-minute dependency guard.
|
||
|
||
|
||
## Re-verified 2026-10-14
|
||
|
||
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
|
||
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
|
||
|
||
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
|
||
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
|
||
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
|
||
|
||
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
|
||
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
|
||
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
|
||
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
|
||
This is the job's **third failure in six days, all three on the opencode path**: 10-08
|
||
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
|
||
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
|
||
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
|
||
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||
|
||
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
|
||
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
|
||
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
|
||
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
|
||
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
|
||
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
|
||
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
|
||
when the LLM turn dies. There is no green-row alternative — the row went red.
|
||
|
||
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
|
||
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
|
||
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
|
||
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
|
||
|
||
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
|
||
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
|
||
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
|
||
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
|
||
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
|
||
|
||
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
|
||
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
|
||
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
|
||
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
|
||
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
|
||
|
||
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
|
||
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
|
||
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
|
||
|
||
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||
session.
|
||
|
||
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
|
||
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
|
||
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
|
||
|
||
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
|
||
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
|
||
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
|
||
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
|
||
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
|
||
gateway runtime env outside the 15-minute dependency guard.
|
||
|
||
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
|
||
|
||
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
|
||
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
|
||
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
|
||
|
||
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
|
||
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
|
||
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
|
||
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
|
||
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
|
||
|
||
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
|
||
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
|
||
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
|
||
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
|
||
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||
|
||
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
|
||
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
|
||
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
|
||
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
|
||
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
|
||
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
|
||
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
|
||
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
|
||
|
||
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
|
||
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
|
||
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
|
||
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
|
||
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
|
||
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
|
||
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
|
||
uncovered.
|
||
|
||
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
|
||
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
|
||
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
|
||
required) and the same on `omniroute:20128` / 100.88.81.19.
|
||
|
||
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
|
||
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
|
||
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
|
||
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
|
||
|
||
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
|
||
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
|
||
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
|
||
→ [[systems/mnemosyne]]
|
||
|
||
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
|
||
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
|
||
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
|
||
|
||
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
|
||
window with no interactive session.
|
||
|
||
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
|
||
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
|
||
Count), `current-state.md`, `index.md`, `log.md`.
|
||
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
|
||
|
||
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
|
||
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
|
||
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
|
||
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
|
||
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.
|
||
|
||
## Re-verified 2026-10-24 (audit window 2026-10-23 00:00 → 2026-10-24 00:3x AWST; live probes 00:1x–00:3x)
|
||
|
||
**Count.** As of 2026-10-24 00:2x AWST (`jobs.json.updated_at 2026-10-24T00:17:36+08:00`): **28 jobs —
|
||
25 enabled / 3 disabled**, unchanged for a sixth consecutive audit. Window **2026-10-23: 169 `completed` /
|
||
0 `failed`** (+ this audit `running`), the third zero-failure day in four; **2026-10-22 settled at 173 / 0**.
|
||
|
||
**The 2026-10-09 timeout storm closed.** `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
|
||
ran clean on its 10-23 slot, closing the storm's last two rows (`1b23c0_c35703f637b8`,
|
||
`1b23c0_103cbe04a79b`) at **08:04**. The storm now holds **zero** unclosed rows and the job reads `ok`
|
||
again → [[incidents/2026-10-09-cron-provider-timeout-storm]].
|
||
|
||
**Incident ledger: unclosed 4 → 2** (no rows opened, two closed). Remaining: `c0a9fa_764558b5ae83`
|
||
(`alerted` since 10-19, next slot Mon 10-26 09:00 — the GitHub Trending Radar revoked token) and
|
||
`0e753e_e7f8194a7d06` (`detected` since 10-01, next slot 11-01 — Facebook Birthday monthly scrape).
|
||
|
||
**Non-clean `last_status`: 5 → 4** — `c0a9fa314542` (`error`, streak 1, Mon 10-26), `0e753e9720fd`
|
||
(`error`, 0, 11-01), `fad5a86979d4` (`delivery_failed`, Mon 10-26), `066bb5191280` (`delivery_failed`, 11-01).
|
||
No new failure signature appeared in the window.
|
||
|
||
**Pins unchanged** — 11 `custom:omni-101` / `smart-route`, 8 unpinned (global default), 6
|
||
`custom:omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1 `custom:omni-101` /
|
||
`deepseek-paid-smart` (this audit).
|
||
|
||
**Sessions: 14 in the window** — 13 `cron` + **1 `desktop`** (2026-10-23 18:26:12, 13 messages, the Conduit
|
||
phone-call session that exposed the memory-timeout fault below).
|
||
|
||
**New fault found and fixed in-window (not a cron job):** the **Conduit** voice plugin gave the Mnemosyne
|
||
provider start a hard-coded **10 s** budget, so three calls (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16)
|
||
were abandoned by the memory layer and ran without recall. Patched the same evening to a **60 s** default
|
||
(`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades instead of aborting; dashboard
|
||
restarted 18:54:43 and **no abort line exists after it**. The patch is **uncommitted** in the plugin's
|
||
working tree → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
|
||
|
||
**Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
|
||
(`250c735`: freshness 2 stale / 94 fresh, skills-index 306).
|
||
|
||
**Pages touched:** `systems/conduit.md` (new), `incidents/2026-10-22-conduit-memory-start-timeout.md` (new),
|
||
`incidents/2026-10-09-cron-provider-timeout-storm.md` (status → resolved), `systems/mnemosyne.md`,
|
||
`current-state.md`, `index.md`, `log.md`, plus frontmatter added to two legacy incident pages (see log).
|
||
**Nothing removed or archived:** no job retired, no system obsolete in the window.
|
||
|
||
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
|
||
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255 stopped;
|
||
the Conduit patch being uncommitted; the 561.7 MB Conduit health store with no retention.
|
||
|
||
## Re-verified 2026-10-25 (audit window 2026-10-24 00:00 → 2026-10-25 00:5x AWST; live probes 00:1x–00:5x)
|
||
|
||
**A third consecutive clean day — 167 / 0 on 2026-10-24, with 2026-10-23 settling at 170 / 0.** Fleet
|
||
unchanged at **28 jobs (25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident
|
||
rows**, pins unchanged. **Nothing opened and nothing closed in the incident ledger in-window.** 14
|
||
sessions: 11 `cron` + 3 `desktop`.
|
||
|
||
**The window's fault is off-cron.** On 2026-10-24 12:53–12:56 Anthony asked three times to be phoned to
|
||
test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`); every reply asserted the
|
||
agent has **no calling capability**, with **zero tool calls** in the session — no `tool_search`, no
|
||
`skill_view`, no `conduit_call_user`. The tool exists, is registered for every session, is reachable
|
||
through lazy discovery, and the plugin ships the `calling-the-user` skill; his call settings
|
||
(`enabled` / `when_asked` / `decides` / `alerts`) were permissive and the device was paired. The first
|
||
reply came from `kilo-auto/free`, but the mid-session switch to `smart-route` repeated it — **capability
|
||
asserted instead of discovered**, not a broken tool →
|
||
[[incidents/2026-10-24-conduit-call-request-refused]].
|
||
|
||
**Conduit re-verified.** Plugin **`0.69.0` / HEAD `f000bf0`** (22 commits on from the `0.64.0` /
|
||
`eaa14be` recorded on 10-24) while the gateway process still dates from **2026-10-23 18:54:43**, so the
|
||
upstream commits are **not live** and the memory-timeout patch is **uncommitted and now five releases
|
||
behind HEAD**. Device phone-status contact last **10-23 19:10:30**; health ingestion ran to
|
||
**10-24 09:59:59**; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** — the 10-24
|
||
page's 308 `buckets` / 62 `windows` was a **narrower grouping**, not a collapse. ⚠ **Evidence
|
||
correction:** `conduit-calls.json → history` is **empty**, and the 10-22 incident's “two calls closed
|
||
cleanly 00:03:23 / 00:06:57” were the two Mnemosyne **cron** turn-ends of that night (`ends` records
|
||
any turn-end that fires a watch, `MAX_RECENT_ENDS = 64`) — so the memory fix still has **no positive
|
||
call evidence** → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/conduit]].
|
||
|
||
**Method note:** daily tallies are bucketed with `substr(started_at,1,10)` — `date(started_at)`
|
||
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day (see the Count section).
|
||
|
||
**Pages touched:** `incidents/2026-10-24-conduit-call-request-refused.md` (new), `systems/conduit.md`,
|
||
`incidents/2026-10-22-conduit-memory-start-timeout.md` (correction), `systems/scheduled-tasks.md`,
|
||
`current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no job retired, no system
|
||
obsolete, no decision reversed in the window.
|
||
|
||
**Still open:** the single-node no-failover topology behind every model-calling job; the GitHub Trending
|
||
Radar token (next run Mon 10-26 09:00); Camofox containment under multi-tab load; Mattermost CT255
|
||
stopped; the Conduit patch uncommitted and **five releases behind HEAD**; the **569.9 MB** Conduit
|
||
health store with no retention; and a **call that has never been placed** since the memory fix.
|
||
|
||
## Re-verified 2026-11-02 (audit window 2026-11-01 00:00 → 2026-11-02 00:4x AWST; live probes 00:0x–00:5x)
|
||
|
||
**Settled: 167 `completed` / 1 `running` / 2 `unknown`** — the first non-clean day after three
|
||
consecutive clean-or-nearly-clean days (10-31 was 161 / 4). Fleet unchanged at **29 jobs, 26 enabled,
|
||
3 disabled**. Nothing was added, removed, disabled or repinned in the window.
|
||
|
||
### The day's three failure modes, none of which is a repeat of 10-31
|
||
|
||
| # | Fault | Jobs | Durable record | Page |
|
||
|---|---|---|---|---|
|
||
| 1 | **`SIGTERM` (`-15`) after adoption, no terminal state** — *"whether side effects ran is unknown"* | `85edf6772bca` 07:30:46, `96eb95ce12f2` + `0e753e9720fd` 09:02:18 | 3 `cron_incidents` rows, all `resolved`; one execution left `unknown` | [[incidents/2026-11-01-cron-worker-sigterm-orphaned-claims]] |
|
||
| 2 | **Gateway handoff failure** — *"Cron execution unverified … Gateway connection failed; no local fallback"* | `b63ed5817212` (self-healed 68 s) and `79f9bb41ecf5` | 2 rows; **`79f9bb_9c443b602eec` still `alerted`** | [[incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure]] |
|
||
| 3 | **`last_status: error` with no execution row and no incident row** | `ecc1fc13655a`, `last_run 2026-11-01T12:01:00` | **none** | same page, "Second silent gap" |
|
||
|
||
Fault 1 is a **fourth variant** of the orphan class (after 10-20 crash, 10-26 drained-`SIGTERM`, 10-31
|
||
cap force-drain) and the first with **no gateway restart anywhere near it** — so it cannot be explained
|
||
by reload/drain behaviour. Fault 3 is the inverse of the usual silent failure: instead of a green row
|
||
over a dead job, there is a **red status with no row at all**.
|
||
|
||
### The 09:00 pair — cold-start cost meets a 30 s ack window
|
||
|
||
At **09:00:49.708** the spawner logged `Cron external worker for job '96eb95ce12f2' did not acknowledge
|
||
within 30s; leaving the durable execution claim untouched`; the worker then spent **09:00:49 → 09:01:02**
|
||
doing a full cold boot (`Plugin discovery complete: 83 found, 69 enabled`) before starting its job at
|
||
09:01:13 and being signalled at ~09:02:13. **The 30 s window is smaller than this host's cold-start
|
||
time.** Not yet formalised as an Open item because the signal source is unidentified — see the incident
|
||
page's open questions.
|
||
|
||
### Delivery health: 8 rows, 7 delivered
|
||
|
||
`deliveries.db` for 11-01: 02:39:33 ✔, 03:00:37 ✔, **07:18:50 ✘** (`066bb5191280`,
|
||
`configured thread_id 34795 for telegram:1793951355:34795 was not found; delivered without thread_id`),
|
||
07:34:49 ✔, 08:36:51 ✔, 10:44:12 ✔, 11:01:12 ✔ — **and nothing after 11:01:12**, which is how the
|
||
missing Weekly Review was detected. The `066bb5191280` case is the **second variant** documented in
|
||
[[systems/messaging-integrations]] (thread-less fallback send, run still marked `failed`).
|
||
|
||
### Incident ledger movement
|
||
|
||
- **Closed on 11-01:** `ea79a2_a4661554a3c2` + `ea79a2_9197df32320b` (02:39:33 — the Mnemosyne Health
|
||
Monitor pair from 10-26 / 10-31) and **`0e753e_e7f8194a7d06`** (10:44:12) — the Facebook birthday
|
||
repin row that had been open since **2026-10-01 09:00:55**, the oldest row on the board. That job's
|
||
`delivery_outcome: delivered` on the same run.
|
||
- **Opened on 11-01:** 5 rows; **4 resolved, 1 open**.
|
||
- **Now unclosed: exactly one** — `79f9bb_9c443b602eec` (`state: alerted`, `acked_at: None`).
|
||
|
||
### Deliberately not done
|
||
|
||
- No full vault mirror pass (drift counts recorded in `log.md`; a complete pass is bigger than an
|
||
audit's mandate).
|
||
- No edits to `### Open` beyond the `066bb5191280` note above: the 10-13 GitHub-token and Camofox
|
||
containment items were not re-probed this window, so their lines are left as last verified.
|
||
- Nothing removed or archived — no job retired, no system obsoleted, no decision reversed.
|
||
|
||
**Still open (carried):** the single-node no-failover topology behind every model-calling job; the
|
||
GitHub Trending Radar token; Camofox containment under multi-tab load; the Conduit health store with no
|
||
retention; **plus new:** the unreconciled admission files
|
||
`~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json` (both 11-01) and the unidentified `-15`
|
||
signal source.
|
||
|
||
## Re-verified 2026-11-03 (audit window 2026-11-02 00:00 → 2026-11-03 00:2x AWST; live probes 00:0x–00:2x)
|
||
|
||
**The first fully clean ledger day in five — and it contains two failures the ledger does not show.**
|
||
|
||
**Settled: 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) against 167 / 1 / 2 on
|
||
11-01, 160 / 4 on 10-31 and 138 / 0 on 10-30. Fleet unchanged at **29 jobs, 26 enabled, 3 disabled**.
|
||
Nothing was added, removed, disabled or repinned in the window. **96** of the completions are
|
||
`96eb95ce12f2`'s 15-minute watchdog slots.
|
||
|
||
**Both 11-01 `unknown` rows resolved in-window** (see §Resolved 2026-11-02). **Unclosed incident rows
|
||
1 → 0** — the first zero in the period. **Non-clean `last_status` 2 → 1.** Deliveries **9 rows, 9/9
|
||
`delivered`.**
|
||
|
||
### Both faults of the day are invisible in the ledger
|
||
|
||
| # | Fault | Job | What the ledger says | Page |
|
||
|---|---|---|---|---|
|
||
| 1 | **Iteration cap reached before the handoff** — 9 h 01 m 19 s, Stage 2 never triggered, **nothing reached Notion**, and **no `cron_incidents` row was raised** | `dca8482e4f76` 07:00:33 → 16:01:52 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | [[incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence]] |
|
||
| 2 | **9 h 43 m run, no report delivered, output left uncommitted** | `1a8e90ea2271` 03:00:31 → 12:44:00 | `completed` / `error: NULL` / `delivery_outcome: suppressed` | this page → Known Issues → Open |
|
||
|
||
Fault 1 is the **third cap in seven days** (10-27 tool-call budget, 10-28 iteration cap, 11-02 iteration
|
||
cap) and the **first that raised no incident at all** — the two earlier rows carry their runs' INCOMPLETE
|
||
reports verbatim in `cron_incidents.error`. That is the sharpest form of the §9 anti-pattern yet: the
|
||
ledger did not merely stay green over a dead job, it **lost the red row it used to produce**. Silence from
|
||
`cron_incidents` is not success.
|
||
|
||
### Delivery health: 9 rows, 9 delivered
|
||
|
||
`deliveries.db` for 11-02: **01:27:21 ✔** (the Weekly Review recovery), 07:30:35 ✔, 07:42:21 ✔,
|
||
08:10:18 ✔, 08:41:14 ✔, 09:07:20 ✔, 10:23:15 ✔, 12:29:25 ✔, 18:45:37 ✔. **The 11-01 `066bb5191280`
|
||
thread-miss did not recur.** That job has **no `executions` row on 11-02 at all** (`0 2 1 * *`, next slot
|
||
2026-12-01), so its 11-01 `delivery_failed` flag will sit on the fleet for the rest of November — the
|
||
fleet's **only** non-clean `last_status`.
|
||
|
||
### Stocktake artefacts committed on the stocktake's behalf
|
||
|
||
`1a8e90ea2271` finished with `systems/skills-index.md` modified and
|
||
`raw/stocktake/freshness-20261102.txt` untracked in the working tree. Both are valid artefacts —
|
||
freshness **4 stale / 113 fresh** at a >21-day threshold; skills-index regenerated with **Devops 108 → 109**
|
||
(`pulse-turn-handling` added) — so this audit **committed both** rather than leave them to be overwritten
|
||
by the next slot.
|
||
|
||
### Method note (correction carried forward)
|
||
|
||
Two rules now bind this audit's tallies: group by `substr(started_at,1,10)`, never `date(...)` (which
|
||
normalises to UTC and shifts 00:00–07:59 AWST rows onto the previous day); and **do not read a
|
||
`running`/`unknown` row as a verdict until the job's expected runtime has elapsed** — the Weekly Review
|
||
was still working at the 00:4x reading and delivered at 01:27, 43 minutes later. An audit that runs two
|
||
hours after midnight will read every long job mid-flight.
|
||
|
||
### Pages touched
|
||
|
||
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (new),
|
||
`incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md` (status → **`resolved`**,
|
||
Resolution section + answers to its open questions 1–3 added), `systems/scheduled-tasks.md`,
|
||
`systems/browser-backend.md`, `systems/skills-index.md` (stocktake output, committed),
|
||
`raw/stocktake/freshness-20261102.txt` (stocktake output, committed), `index.md`, `log.md`.
|
||
**Nothing removed or archived:** no job retired, no system obsolete, no decision reversed in the window.
|
||
|
||
### Still open (carried + new)
|
||
|
||
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
|
||
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
|
||
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the
|
||
unidentified `-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**.
|
||
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
|
||
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
|
||
|