Files
agent-estate-wiki/systems/scheduled-tasks.md
T
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00

479 lines
57 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Scheduled Tasks
type: system
status: active
created: 2026-07-22
updated: 2026-10-16
verified_on: 2026-10-16
review_after: 2026-10-25
confidence: high
tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db]
---
# Scheduled Tasks
## Count
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10)
- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule.
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
- **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
- **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs.
- **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**.
- **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`).
- **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`).
- **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice.
- **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day.
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12)
- **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive.
- **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart).
## Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 |
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) |
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it |
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 |
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken |
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
## Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
## Known Issues
### Resolved 2026-10-12
- **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
- **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]].
- **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot).
### Resolved 2026-10-07
- **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
- **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).
### Resolved 2026-10-05
- **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]].
- The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run.
- **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.**
- **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
### Resolved 2026-09-30
- **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]].
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0.
- **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below).
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
### Open
- **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`.
- **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]].
- **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
- **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.
- **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
- **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references.
- **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:<topic>`. See [[systems/messaging-integrations]].
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
### Historical
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
## Related
- [[systems/mnemosyne]]
- [[runbooks/job-radar]]
- [[systems/hermes-agent]]
- [[systems/messaging-integrations]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
---
## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)
**Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at
2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day:
**10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}.
**The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one
`RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST;
every run from 18:00 onward completed. Full timeline, job list and incident table:
[[incidents/2026-10-09-cron-provider-timeout-storm]].
- Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05,
`ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`.
- `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its
03:00 run today decides whether the streak clears.
- `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in
`agent.log`, so it reported the storm rather than suffering it.
**The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed
(`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and
committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08
00:12). The 2026-10-09 run covers both days.
**Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit
step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator
ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output.
**DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26
DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*.
Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`,
`last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel
(`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not
mutate cron state**; retiring the job is open item #1 below.
**Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09;
capture and vector health were verified good (19 new working rows since 10-08; 760/760 working,
1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**.
See [[incidents/2026-10-09-memory-recall-relevance]].
### Open items (2026-10-09)
1. ~~**Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.~~ **DONE 2026-10-10** — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See [[log.md]] and the 2026-10-10 audit block below.
2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test.
3. ~~**Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).~~ **DONE** — all eight cleared: five on 2026-10-10 and `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41. (A different pair, `1b23c0_c35703f637b8` + `0e753e_e7f8194a7d06`, remains open for unrelated reasons — see `### Open` above.)
4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit.
5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open.
6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.
---
## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)
**Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`).
The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**:
open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail
monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` -
10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was
wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.
**Non-clean `last_status`: 5 jobs, down from 10.**
| Job | ID | `last_status` | streak |
|---|---|---|---|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 |
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 |
| Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 |
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 |
| Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 |
The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271`
(stocktake) is back to `last_status: ok`.
**Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean
(`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc`
12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen
2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30).
**Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening
of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID
lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted"
claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was
the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.
**Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index
off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost
platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
---
## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)
**Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled
set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher),
`0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy).
**Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` —
10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still
`running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive
zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard
`96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor
`b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1.
**Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0),
`fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1),
`066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers:
their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has
had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should
clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`).
**Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3:
`fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400),
`dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from
the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted),
`0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 /
closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.
**Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51,
41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule
(`completed` 03:00:40 → 03:11:29) and committed `d981e30`.
**False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a
capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST
`now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the
session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned
`[SILENT]` (green). → [[systems/mnemosyne]]
**Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`);
skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows;
Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the
15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]).
## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)
**Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed —
and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on
paper had its delivery hop dead the whole time. Detail below.
**Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit
finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit
itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention
floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer
reproducible from this DB).
**Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at
**03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`);
`dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals
now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06`
(`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and
`1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**).
**Job-by-job (verified against the executions log, `delivery_outcome` included):**
- `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**.
Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof.
- `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030
(32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy —
see the Camofox finding — but it produced and delivered.
- `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its
own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the
job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.
- `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed**
(`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00.
- `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window**
(monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed.
**Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the
previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140),
`1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and
pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on
disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step
skipped on the documented path.
**Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 =
99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly
(`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows;
1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working /
1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to
act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures.
**Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the
`telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401
Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed,
and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending
pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the
Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and
its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has
been corrected: two green runs were read as proof the token worked, when a green cron row only means the
*agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value
not recorded here.
**Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of
`camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G,
swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom →
`/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and
completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented
(`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]]
**Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page
archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).
**Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs.
multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean
`last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-14
Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**.
**Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row
in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0`
→ **the first failed cron run since 2026-10-09** (a day that recorded 15).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.**
03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.`
Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`,
`failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule).
This is the job's **third failure in six days, all three on the opencode path**: 10-08
`HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout
(`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on
2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on
the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts; this audit committed them.**
The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk,
then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree:
`systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit.
Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing
when the LLM turn dies. There is no green-row alternative — the row went red.
**Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight);
`MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**.
Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its
finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.
**Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar
(`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`),
Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing
(`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread
delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01.
**Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding
Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and
no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare;
the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is
tolerant of wrapping rather than literal. → [[systems/mnemosyne]]
**Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success
10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09),
`066bb5191280` (10-01), `1a8e90ea2271` (10-13).
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note),
`systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`.
**Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged.
**Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the
10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode
or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run
Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert;
skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable;
gateway runtime env outside the 15-minute dependency guard.
## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm
Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the
same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x.
Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`).
**The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed /
24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`,
10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on
**12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not
fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).
**Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`**
(`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.`
→ `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations.
Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
**Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18
enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is
vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on
`custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**.
⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is
not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.**
**Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h),
`1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action
Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`,
`0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident
page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political
News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth
New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went
uncovered.
**Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak
4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16
**00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth
required) and the same on `omniroute:20128` / 100.88.81.19.
**Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`,
`ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above.
Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280`
(10-01 `delivery_failed`, dead Telegram `thread_id 34795`).
**Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the
window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12.
→ [[systems/mnemosyne]]
**Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt`
was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00,
so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run.
**Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new),
`systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter,
Count), `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page.
**Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one
OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's
call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab
workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows;
Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.