Files
agent-estate-wiki/systems/scheduled-tasks.md
T
hermes 2061b8607b audit(2026-10-16): fleet-wide 34h cron blackout 10-14 -> 10-15 — omniroute-1 refused connections
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).

STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.

New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
2026-10-16 00:19:14 +08:00

57 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Scheduled Tasks system active 2026-07-22 2026-10-16 2026-10-16 2026-10-25 high
cron
scheduled
automation
live-cron-list
live-jobs-json
cron-executions-db

Scheduled Tasks

Count

  • ⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery. 12 non-clean last_status (was 3 at the 10-13 audit): 9 new rows from the omni-101 outage — 08eb86f02d37 (streak 2), ecc1fc13655a (2), dca8482e4f76, c8d52904e1d6, 06a0a1c8704f, 85edf6772bca, 0e519874d280, 1a8e90ea2271, cbfe434a3b5e (all streak 1) — plus the three long-standing rows 1b23c042e17d (since 10-09), 0e753e9720fd (10-01) and 066bb5191280 (delivery_failed, 10-01). Three jobs already cleared inside this window: b63ed5817212 (10-16 00:02), 1b31187397a5 (10-15 22:01), ea79a2107e36 (10-16 00:09). Structural finding: all 18 enabled model-calling jobs are pinned to custom:omni-101 — the cron fleet has no model-call failover. → incidents/2026-10-14-omniroute-node-connection-refused-storm
  • Total jobs: 29 — 26 enabled, 3 disabled (verified from live ~/.hermes/cron/jobs.json, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor 62619aa6bcb3 was retired on 2026-10-10)
  • Non-clean last_status: 3 (was 5 at the 2026-10-12 audit) — 0e753e9720fd (error, streak 0), 1b23c042e17d (error, streak 1), 066bb5191280 (delivery_failed, streak 0). fad5a86979d4 and dca8482e4f76 cleared on 2026-10-12 when both ran clean on schedule.
  • Failing (last run failed outright): 0 — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider custom:omni-101 using routing aliases, and every failure_streak in jobs.json now reads 0. Four of the seven jobs had still carried a pre-repin last_status: error (1b23c042e17d, fad5a86979d4, 0e753e9720fd, 1a8e90ea2271) — a stale field, not a live failure; it clears on each job's next scheduled run. 1a8e90ea2271 cleared on 2026-10-06 03:00, running on schedule for the first time since 2026-09-25; the remaining three (1b23c042e17d, fad5a86979d4, 0e753e9720fd) are due Fri 08:00 / Mon 03:00 / 1st 09:00. fad5a86979d4 cleared on 2026-10-12 03:00, leaving two (1b23c042e17d, 0e753e9720fd). See incidents/2026-10-05-dead-model-pin-repin.
    • Historical (2026-10-01 → 2026-10-05), now all remediated: dca8482e4f76 403 MODEL_NOT_IN_PLAN ×7; 1a8e90ea2271 HTTP 410 ×11 (retired qwen3.5:397b); fad5a86979d4 HTTP 400 ×3; 0e753e9720fd HTTP 400 ×1; 06a0a1c8704f, 1b23c042e17d and 79f9bb41ecf5 HTTP 401 (the cliprox/terra connection auth expiry, settled 3/3 on 2026-10-04).
  • Failing (delivery only): 1 — 066bb5191280: the report itself generated; the send failed against the DM thread (bare telegram target, 2026-10-01). Unchanged this window.
  • Repin applied 2026-10-05: free-stack ×6 (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd) and smart-route ×1 (dca8482e4f76), all on custom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. free-stack is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs.
  • Verified live, not assumed: 79f9bb41ecf5 succeeded on a manual run 2026-10-05 23:39 and 06a0a1c8704f at 23:45 (both incidents auto-resolved); dca8482e4f76 completed 23:56 and delivered to telegram:-1004321904721:1030 (incident resolved 2026-10-06 00:05:07); and 1a8e90ea2271 completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d, fad5a86979d4, 0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules. 1a8e90ea2271 has since run twice more on schedule (2026-10-06 03:00 → 33f4e95; 2026-10-07 03:00 → 8c63ef2), so the stocktake repin is verified across consecutive automatic runs.
  • Streak counting note: the N failures in a row figure in hermes cron list comes from the failure_streak counter in jobs.json. executions.db retains only rows from 2026-09-30 onward, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: all 0.
  • Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled). 066bb5191280 and 0e753e9720fd still carry the bare-telegram target warning (configured thread_id 34795 for telegram:1793951355 was not found).
  • Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30 failure_streak counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 completed, 0 failed, 1 running (the audit itself); 14 sessions, all cron — no interactive (non-cron) sessions. The stocktake job 1a8e90ea2271 ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plus raw/stocktake/freshness-20261006.txt, committing to Gitea as 33f4e95 — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin last_status: error (1b23c042e17d, fad5a86979d4, 0e753e9720fd).
  • A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a 85edf6772bca failure notice that had been sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at (and the cron_incidents state) before acting on a notice.
  • Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00): job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — every failure_streak reads 0. Window tally (executions.db): 176 completed, 1 failed, 1 running (the audit itself). The single failure is 62619aa6bcb3 (DHL Tracking, 08:00) — a Camofox HTTP read timeout (ensure_browser() → TimeoutError: timed out), notified 08:00:58 and auto-resolved 11:01:13 when the next 3-hourly run succeeded (incident 62619a_f485ccfd4a0d). 13 sessions started in the window, all source = cron — no interactive sessions; 12 deliveries, 0 delivery failures. The same three stale pre-repin last_status: error jobs remain (1b23c042e17d, fad5a86979d4, 0e753e9720fd); the stocktake job 1a8e90ea2271 ran on schedule at 03:00 for the second consecutive day.
  • Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07) → 30, 0 failing (2026-10-08) → 29, 0 failing (2026-10-10) → 29, 0 failing (2026-10-11) → 29, 0 failing (2026-10-12)
  • Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00): 29 jobs (26/3), 0 failing, 5 non-clean last_status; window tally 172 completed, 0 failed (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 running). 18 sessions — 17 cron + 1 interactive.
  • Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00): 29 jobs (26/3), 0 failing, non-clean last_status down to 3. Window tally 174 completed, 0 failed — third consecutive clean day (10-10 175/0, 10-11 172/0, 10-12 174/0). 5 open incidents → 2: three closed on 2026-10-12 (see #Resolved 2026-10-12). 18 sessions in the window, all cron, no interactive sessions; 0 new incident pages required. Two findings led the window: the GitHub Trending Radar delivery token is dead (getMe → 401, live-verified) and the Daily Marketing Job Radar run hit Camofox memory pressure (2.5 GB peak, heavy swap, self-recovered by restart).

Job Table

Name ID Schedule Delivery Skills Script Notes
AgentMail Rhino Inbox Poll 0a7f933d06e2 * * * * * origin terminal agentmail-poll.py disabled since 2026-07-27 (legacy)
AI Newsletter Curation for Patrick (Weekly) 06a0a1c8704f 0 10 * * 4 telegram:-1004321904721:346 — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0
Daily Birthday Checker a432898b55a4 30 7 * * * telegram:1793951355 — birthday_check.py
Daily Marketing Job Radar dca8482e4f76 0 7 * * 1-5 telegram:-1004321904721:1030 job-search-automation — ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). ✅ First scheduled success on the repin 2026-10-12 07:00 (finished 07:33:50, delivered) — its 2026-10-09 timeout incident dca848_23fcd2b18aaa auto-closed 07:33:41. Live last_status: ok, streak 0. ⚠ The 2026-10-12 run hit Camofox memory pressure — the service reached its containment ceiling (2.5 GB peak, MemoryHigh=2G, swap 1 GB fully consumed, available: 0B), so /tabs calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open)
DHL Tracking - Parcel 8052838382 62619aa6bcb3 0 8,11,14,17,20 * * * telegram:1793951355 — dhl_tracking_monitor.py no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (ensure_browser() → TimeoutError: timed out) — incident 62619a_f485ccfd4a0d, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 Delivered — the parcel has arrived, so the monitor is now redundant
Enhanced Weekend Adventure Planner 1b23c042e17d 0 8 * * 5 telegram:-1004321904721:102 — — 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). ⚠ Last run 2026-10-09 08:00 failed — RuntimeError: Request timed out. (incident 1b23c0_c35703f637b8, still alerted, opened 2026-10-09 08:17) — so it still carries last_status: error, streak 1. Next scheduled slot: Fri 2026-10-16 08:00, which should clear it
Evening Stand-Down Briefing e530c89f2be0 45 18 * * 1-5 telegram:-1004321904721 — evening_standdown.py no-agent
Facebook Birthday Monthly Scrape 0e753e9720fd 0 9 1 * * telegram — — ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident 0e753e_e7f8194a7d06 (opened 2026-10-01 09:00:55) is still open in state detected, and last_status still reads error; next slot 2026-11-01 09:00, which is when the repin finally gets its live test. ⚠ Bare telegram still targets the home DM — retarget to a group topic
Facebook Birthday Weekly Registry Sync fad5a86979d4 0 3 * * 1 telegram — birthday_check.py ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). ✅ Verified live 2026-10-12 03:00 — completed and delivered (448 birthdays); the stale last_status: error cleared after two weeks and both deepseek-v4-flash incidents auto-closed at 03:05:39 (fad5a8_a13bbc837e1c, fad5a8_c80b59a907f0). Live last_status: ok, streak 0. ⚠ Bare telegram target should move to a group topic
FreshRSS Midday News Briefing ecc1fc13655a 0 12 * * * telegram:-1004321904721:346 — freshrss_intelligence_v2.py
GitHub Trending Radar — Weekly Roundup c0a9fa314542 0 9 * * 1 telegram:1793951355 — — ⚠ Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13). A getMe call against the telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 Unauthorized. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at digest.py's send(). The agent recovered the content with --dry-run and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new @BotFather token replaces the dead one. last_status: ok because the 401 is inside the script, not the cron run. This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.
Gmail Action Digest 08eb86f02d37 30 8 * * * telegram:-1004321904721:notifications — gmail_action_digest_v3.py
Important-mail monitor d40085da631c every 30m telegram:1793951355 — _cron_silent_monitor.py disabled since 2026-08-29 (Gmail read-state incident)
Job Tracker Dashboard Refresh 2fd50c87b0d0 0 */2 * * * telegram:-1004321904721:1030 — dashboard_watchdog.py no-agent
Mnemosyne Auto-Consolidate 0c771463c3cd every 30m local — consolidate_mnemosyne.sh no-agent
Mnemosyne DB Backup e5ed215fc91c 0 3 * * 0 telegram:1793951355 — mnemosyne-backup.sh no-agent
Mnemosyne Embedding Coverage Monitor b63ed5817212 0 */6 * * * telegram:-1004321904721:102 — check_embedding_coverage.py Alerts if coverage < 95%
Mnemosyne Health Monitor ea79a2107e36 0 */12 * * * telegram:-1004321904721:102 mnemosyne-operations — Reports only on exception
Mnemosyne Watchdog 96eb95ce12f2 */15 * * * * telegram:-1004321904721:102 — mnemosyne-watchdog.py no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30)
Mosquito Tracker 1b31187397a5 0 7,15,22 * * * telegram:-1004321904721:273 — mosquito_tracker.py Excludes 23:00–07:00
Perth New Restaurant & Cafe Openings (Weekly) c8d52904e1d6 0 18 * * 4 telegram:-1004321904721:102 — — delivery target corrected 2026-09-30; verified delivered, status ok
Political News Digest (Mon/Thu) 0e519874d280 0 8 * * 1,4 telegram:-1004321904721:346 — —
Rhino AgentMail Inbox Watcher f77b7e46bd09 every 5m origin — agentmail-rhino-watch.sh disabled since 2026-07-27 (legacy)
Tech & AI Newsletter Digest (Analytical) 85edf6772bca 0 7 * * * telegram:-1004321904721:346 — —
Uber Eats Monthly Spend Report 066bb5191280 0 2 1 * * telegram camofox-browser-automation, browser-backend-verification — ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic
Weekly Review 79f9bb41ecf5 0 18 * * 0 origin weekly-review — ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0
weekly-save-restricted-bot-restart a688ed7b7ab0 0 3 * * 0 local — restart-save-restricted-bot.sh no-agent; script restored 2026-09-30
Wiki Audit — Nightly cbfe434a3b5e 59 23 * * * local task-approach — Reviews past 24h for wiki changes
wiki-stocktake-maintenance 1a8e90ea2271 0 3 * * * local — wiki-maintain.sh ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ The Quartz rebuild step is a no-op in practice — no quartz.config.ts exists and npx is unavailable, so the static site in public/ is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit 8c63ef2) — a second consecutive automatic success, verified. The script itself was never broken
🦝 Hermes Config Backup to Gitea (Daily) e27da6c34bbb 0 11 * * * origin — hermes-config-backup.sh no-agent

Removed Jobs

Name Removed Reason
Workday start reminder 2026-08-26 No longer present in live cron list
CT 221 Miner Watchdog 2026-08-26 Job d5516d0a5147 removed; CT221 status unknown/inspected
Review Agent Estate Phase One 2026-08-26 One-time job; completed 2026-08-08, now cleaned up

Disabled Jobs (not removed)

Name ID Disabled since Reason
Important-mail monitor d40085da631c 2026-08-29 Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state.
Rhino AgentMail Inbox Watcher f77b7e46bd09 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.
AgentMail Rhino Inbox Poll 0a7f933d06e2 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.

Known Issues

Resolved 2026-10-12

  • Both deepseek-v4-flash incidents closed by a clean scheduled run (fad5a86979d4, 03:00): the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents fad5a8_a13bbc837e1c (open since 2026-09-28, every fallback provider exhausted) and fad5a8_c80b59a907f0 (since 2026-09-30, deepseek-v4-flash HTTP 400) both auto-closed at 03:05:39. Its stale pre-repin last_status: error cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one.
  • dca8482e4f76 timeout incident closed (dca848_23fcd2b18aaa): the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm auto-closed at 07:33:41. Of the ten storm incidents, nine have now closed and one remains: 1b23c0_c35703f637b8 (1b23c042e17d, still alerted, next slot Fri 2026-10-16 08:00). See incidents/2026-10-09-cron-provider-timeout-storm.
  • Net effect: open cron incidents 5 → 2. The two survivors are 0e753e_e7f8194a7d06 (detected since 2026-10-01, untouched because its job does not run until 2026-11-01) and 1b23c0_c35703f637b8 (alerted, cleared on the next Friday slot).

Resolved 2026-10-07

  • DHL Tracking 62619aa6bcb3 — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01): the 08:00 run died in ensure_browser() with TimeoutError: timed out on a browser HTTP call (dhl_tracking_monitor.py:233), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (62619a_f485ccfd4a0d, failure_type: timeout) auto-resolved at 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script.
  • The tracked parcel arrived. The 14:00 run on 2026-10-07 reported waybill 8052838382 Delivered (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire).

Resolved 2026-10-05

  • Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream — cliprox/terra (HTTP 401, connection auth expired, 3 jobs), ollama-cloud/qwen3.5:397b (HTTP 410, retired 2026-09-25), nvidia/deepseek-ai/deepseek-v4-flash (HTTP 400, dropped from the live catalog, 2 jobs) and command-code/gpt-5.6-terra (HTTP 403 MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider custom:omni-101 using routing aliases — free-stack ×6, smart-route ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. failure_streak is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: 79f9bb41ecf5 (23:39), 06a0a1c8704f (23:45) and dca8482e4f76 (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale last_status: error field clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.
    • The cliprox/terra incident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See incidents/2026-10-01-cliproxy-connection-auth-expired.
    • wiki-stocktake-maintenance resumes its 03:00 schedule, so wiki-maintain.sh no longer needs a daily manual run.
  • GitHub Trending Radar c0a9fa314542 — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13. The Mon 09:00 run completed on 2026-10-05 and 2026-10-12, which was read as proof the getMe → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live getMe against the token in /home/hermes/workspace/github-trending-radar/config.json returns 401. Corrected above and in Open items. Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.
  • A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a 85edf6772bca failure notice sitting in topic 346 since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already resolved on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.

Resolved 2026-09-30

  • Telegram thread delivery failures — root cause found and fixed: jobs fad5a86979d4, 1b23c042e17d, c8d52904e1d6 reported configured thread_id 34795 for telegram:1793951355 was not found / Message thread not found. The target was wrong, not the topic: a bare deliver: "telegram" resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/messaging-integrations.
  • Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause was ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2, run every 15 min by job 96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • weekly-save-restricted-bot-restart missing script (2026-08-30): Previously failed with "Script not found: restart-save-restricted-bot.sh". The script existed at /home/hermes/scripts/ but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now ok, streak 0.
  • Residual pre-fix dispatch errors cleared: cbfe434a3b5e returned to ok on its next run as predicted. The 1a8e90ea2271 error of the same era also cleared — its current failure is a different, upstream problem (below).
  • Stale delivery_failed flag on 1b23c042e17d cleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic 102.

Open

  • GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13): live getMe on the token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401, so digest.py's Telegram send fails every Monday while the cron run itself stays green (last_status: ok). A completed run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with --dry-run and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in config.json, and re-run getMe to confirm; do not re-read a green cron status as proof of delivery. This corrects the 2026-10-05 "claim refuted" entry, which was wrong. See systems/scheduled-tasks job c0a9fa314542.
  • Camofox memory pressure during long browser runs (NEW 2026-10-13): the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf: MemoryHigh=2G, MemoryMax=3G, swap max 1G). Live probe mid-run read Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G) with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the /tabs endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See systems/browser-backend.
  • Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07): wiki-maintain.sh documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no quartz.config.ts is present and npx is unavailable, so public/ has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description.
  • skills-index under-counts by one skill (NEW 2026-10-07): live find ~/.hermes/skills -name SKILL.md -type f returns 347, but the generated page says 346 (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md collapses into its parent and is silently dropped. The other duplicated basename (hermes-api-vault, at the root and under devops/) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename.
  • A delivered notice is not a successful job: cliprox/terra failures recorded delivery_outcome: delivered because the failure notice is itself a deliverable. Read the execution status column, not the delivery outcome, when judging whether a job produced its output.
  • Stale provider key custom:omni (1 enabled + 1 disabled): the provider was removed from custom_providers (live set on 2026-10-01: litellm, guacano, Omniroute:20128, command-code, omni-101). Re-verified 2026-10-06: e530c89f2be0 (enabled) and d40085da631c (disabled) still name it; both run smart-route, and the default custom provider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised to custom:omni-101. The 2026-10-05 repin moved seven jobs to an explicit custom:omni-101, leaving these two as the only remaining custom:omni references.
  • Bare telegram delivery targets (3 jobs) — demonstrably breaking sends: 0e753e9720fd, fad5a86979d4, 066bb5191280 deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with configured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three to telegram:-1004321904721:<topic>. See systems/messaging-integrations.
  • Two wiki trees, drifted in both directions (NEW 2026-10-02): /home/hermes/workspace/agent-estate-wiki is the canonical source (Gitea remote Anthony/agent-estate-wiki, Quartz build, wiki-maintain.sh target). /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads — pages nested under wiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md, systems/nextcloud.md, three incidents/ pages exist only in the vault; raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md only in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent.
  • Work mailbox unreachable (non-cron; affects on-demand lookups): the work Outlook account is configured with password auth while the tenant advertises only XOAuth2, so himalaya -a work fails with login not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.

Historical

  • Mnemosyne Watchdog incident (2026-08-19): Job 96eb95ce12f2 cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne.
  • Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
  • Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to smart-route or gpt-5.6-terra.
  • Wiki Audit model drift (2026-08-07): Global inference config change caused RuntimeError: Skipped to prevent unintended spend on an unpinned job.

Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09)

Fleet: 30 jobs · 27 enabled · 3 disabled · 10 carrying a non-clean last_status at 2026-10-10 00:22 (jobs.json.updated_at 2026-10-10T00:22:17+08:00). Executions by AWST day: 10-07 {'completed': 177, 'failed': 1} · 10-08 {'completed': 178, 'failed': 2} · 10-09 {'completed': 162, 'failed': 15, 'running': 1}.

The 2026-10-09 timeout storm. 16 failed executions across 11 distinct jobs, every one RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, 00:00 → 15:00 AWST; every run from 18:00 onward completed. Full timeline, job list and incident table: incidents/2026-10-09-cron-provider-timeout-storm.

  • Four incidents auto-closed on the next clean run (96eb95 07:01, b63ed5 18:01, 1b3118 22:05, ea79a2 2026-10-10 00:03); eight remain open — 1a8e90_789224ad589c, cbfe43_fa81052baaf7, 1a8e90_dc6b2d40ec44, dca848_23fcd2b18aaa, 85edf6_18bedbeec5cd, 1b23c0_c35703f637b8, 08eb86_634f1a296f78, ecc1fc_58b6c38296f3.
  • 1a8e90ea2271 is the only job still at failure_streak: 2 (10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears.
  • 96eb95ce12f2 (watchdog) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, so it reported the storm rather than suffering it.

The 2026-10-08 window has no audit entry. The nightly audit's own 23:59 run on 2026-10-08 failed (RuntimeError: Request timed out., incident cbfe43_fa81052baaf7, first seen 2026-10-09T00:32:09) and committed nothing; the last committed audit covers the 2026-10-07 window (commit 659356a, 2026-10-08 00:12). The 2026-10-09 run covers both days.

Stocktake: two-day gap, artifacts recovered. The 2026-10-08 03:01 run left systems/skills-index.md modified and raw/stocktake/freshness-20261008.txt untracked; both were stranded (the job's own commit step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator ran, so no freshness-20261009.txt exists and the skills index is still the 2026-10-08 output.

DHL monitor — retirement instructed, still enabled. Anthony replied "Delete this job" to the 20:26 DHL delivery notice on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". Live jobs.json at 2026-10-10T00:22 shows 62619aa6bcb3 still enabled — paused_at: null, last_status: ok, schedule 0 8,11,14,17,20 * * * — so it runs again at 08:00. The parcel (8052838382) was reported Delivered on 2026-10-07, making the monitor redundant. This audit did not mutate cron state; retiring the job is open item #1 below.

Memory recall relevance. "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is not in live config. See incidents/2026-10-09-memory-recall-relevance.

Open items (2026-10-09)

  1. Retire 62619aa6bcb3 (DHL Tracking — Parcel 8052838382) — instructed 2026-10-09 21:24, not executed. DONE 2026-10-10 — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See log.md and the 2026-10-10 audit block below.
  2. Decide and persist the recall-ranking change in memory.mnemosyne; re-run the trigger question as a regression test.
  3. Close the eight open cron incidents once their next clean runs land; watch 1a8e90ea2271 (streak 2). DONE — all eight cleared: five on 2026-10-10 and dca848_23fcd2b18aaa on 2026-10-12 07:33:41. (A different pair, 1b23c0_c35703f637b8 + 0e753e_e7f8194a7d06, remains open for unrelated reasons — see ### Open above.)
  4. Stale delivery targets — 0e753e9720fd, fad5a86979d4 (last_status: error) and 066bb5191280 (delivery_failed) unchanged since the 2026-10-07 audit.
  5. Quartz site rebuild is skipped by wiki-maintain.sh step 3 (no quartz.config.ts, no npx) — carried from 2026-10-07, still open.
  6. Skills index off by one (347 SKILL.md on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open.

Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST)

Fleet: 29 jobs - 26 enabled - 3 disabled (jobs.json.updated_at 2026-10-11T00:19:44+08:00). The count fell from 30 to 29 because 62619aa6bcb3 (DHL Tracking - Parcel 8052838382) is gone: open item #1 of the 2026-10-09 audit is CLOSED (below). Disabled: d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll).

Executions by AWST day (executions.db): 10-07 177 completed / 1 failed - 10-08 178 / 2 - 10-09 163 / 15 - 10-10 174 completed / 0 failed (+1 left running). The 2026-10-10 fleet was wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm.

Non-clean last_status: 5 jobs, down from 10.

Job ID last_status streak
Facebook Birthday Monthly Scrape 0e753e9720fd error 0
Facebook Birthday Weekly Registry Sync fad5a86979d4 error 0
Daily Marketing Job Radar dca8482e4f76 error 1
Enhanced Weekend Adventure Planner 1b23c042e17d error 1
Uber Eats Monthly Spend Report 066bb5191280 delivery_failed 0

The three streak 0 rows are stale last_status fields, not live failure streaks. 1a8e90ea2271 (stocktake) is back to last_status: ok.

Incidents: 7 closed during 2026-10-10, 5 open. The 10-09 storm unwound as each job next ran clean (ea79a2 00:03, cbfe43 00:27, 1a8e90ea2271 x2 03:06, 85edf6 07:04, 08eb86 08:32, ecc1fc 12:05). Still open: 1b23c042e17d + dca8482e4f76 (timeout, alerted, both first seen 2026-10-09) and 0e753e9720fd + fad5a86979d4 (agent; open since 2026-10-01 / 2026-09-30).

Open item #1 CLOSED - the DHL monitor is retired. Anthony instructed removal again on the evening of 2026-10-10 and the job is absent from live jobs.json, verified 2026-10-11 00:19 by direct ID lookup (no job id contains 62619; no job name matches dhl/parcel). This is the third "deleted" claim for this job - the 2026-10-07 and 2026-10-09 reports were both false because the check was the absence of job output rather than a positive lookup. Removals must be verified by ID lookup.

Still open: the Quartz rebuild step in wiki-maintain.sh (no quartz.config.ts) - skills-index off-by-one (347 SKILL.md vs 346 reported) - the 5 non-clean last_status rows above - Mattermost platform unreachable (systems/messaging-integrations) - the gateway runtime env is outside the 15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).


Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x)

Fleet: 29 jobs — 26 enabled / 3 disabled (jobs.json updated_at 2026-10-12T00:10:42+08:00). Disabled set unchanged: d40085da631c (Important-mail monitor), f77b7e46bd09 (Rhino AgentMail Inbox Watcher), 0a7f933d06e2 (AgentMail Rhino Inbox Poll — legacy).

Executions by AWST day (executions.db): 10-07 177 completed / 1 failed — 10-08 178 / 2 — 10-09 163 / 15 — 10-10 175 completed / 0 failed (finalised; the 10-11 audit saw 174 + 1 still running) — 10-11 171 completed / 0 failed + 1 running (this audit). Two consecutive zero-failure days after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard 96eb95ce12f2 96 runs (15-min cadence), 0c771463c3cd 46, 2fd50c87b0d0 12, coverage monitor b63ed5817212 3, health monitor ea79a2107e36 2, stocktake 1a8e90ea2271 1.

Non-clean last_status: 5 jobs — unchanged from 2026-10-10. 0e753e9720fd (error, streak 0), fad5a86979d4 (error, streak 0), dca8482e4f76 (error, streak 1), 1b23c042e17d (error, streak 1), 066bb5191280 (delivery_failed, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers: their last_error is RuntimeError: Request timed out. from Friday 2026-10-09 (the storm) and neither has had a scheduled slot since — dca8482e4f76 next runs 07:00 Mon, 1b23c042e17d Fri 08:00, so both should clear themselves. Everything else reads last_status: ok (this build reports ok, not success).

Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged. alerted 3: fad5a8_c80b59a907f0 (weekly registry sync, scope agent, first seen 2026-09-30, deepseek-v4-flash HTTP 400), dca848_23fcd2b18aaa + 1b23c0_c35703f637b8 (timeout, both first seen 2026-10-09 — the last two left from the storm). detected 2: fad5a8_a13bbc837e1c (2026-09-28, every provider in the fallback chain exhausted), 0e753e_e7f8194a7d06 (2026-10-01, same deepseek-v4-flash 400). Table totals: alerted 3 / detected 2 / closed 8 / resolved 63. Last closures remain the seven on 2026-10-10.

Sessions in the window: 18 — 17 cron + 1 interactive (talaria-ios, 2026-10-11 19:13 → 19:51, 41 messages, "best VNC clients" research). The 03:00 stocktake 1a8e90ea2271 ran clean on schedule (completed 03:00:40 → 03:11:29) and committed d981e30.

False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a capture stall. The run compared a naive-UTC working_memory.created_at (2026-10-10 13:24:53) against AWST now and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned [SILENT] (green). → systems/mnemosyne

Still open (unchanged): Quartz rebuild step inert (wiki-maintain.sh step 3 — no quartz.config.ts); skills-index off-by-one (347 SKILL.md on disk vs 346 reported); the 5 non-clean last_status rows; Mattermost platform unreachable (systems/messaging-integrations); gateway runtime env outside the 15-minute dependency guard (incidents/2026-09-28-hermes-managed-runtime-deps).

Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45)

Headline: a clean run hid a dead credential. The cron table was green — 174 completed / 0 failed — and two of the three non-clean last_status rows cleared overnight, yet one job that reported success on paper had its delivery hop dead the whole time. Detail below.

Cron tally: 2026-10-11 172 completed / 0 failed (the one row still running at the last audit finished clean); 2026-10-12 174 completed / 0 failed / 1 running — and the running row is this audit itself (cbfe434a3b5e, started 23:59:41). Third consecutive clean day. The execution-log retention floor has moved up to 2026-10-07 08:15, so 10-07 rows are now pruned (their earlier tally is no longer reproducible from this DB).

Incidents: open 5 → 2. Three closed on 2026-10-12 — fad5a8_a13bbc837e1c and fad5a8_c80b59a907f0 at 03:05:39 (both deepseek-v4-flash, closed by the first clean scheduled run of fad5a86979d4); dca848_23fcd2b18aaa at 07:33:41 (closed by dca8482e4f76's own successful 07:00 run). Table totals now alerted 1 / detected 1 / closed 8 / resolved 66 (76 rows). Survivors: 0e753e_e7f8194a7d06 (detected since 2026-10-01 — cannot clear until its job next runs on 2026-11-01) and 1b23c0_c35703f637b8 (alerted since 2026-10-09, next slot Fri 2026-10-16 08:00).

Job-by-job (verified against the executions log, delivery_outcome included):

  • fad5a86979d4 — Facebook Birthday Weekly Registry Sync 03:00:45 → 03:05:47, completed, delivered. Output: Birthday Dashboard, 448 unique birthdays, 1 contact match. This is the repin's live proof.
  • dca8482e4f76 — Daily Marketing Job Radar 07:00:47 → 07:33:50, completed, delivered to topic 1030 (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — see the Camofox finding — but it produced and delivered.
  • c0a9fa314542 — GitHub Trending Radar 09:00:41 → 09:08:33, completed, delivered — and yet its own send failed (finding 1). A completed row plus delivery_outcome: delivered is not proof the job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's.
  • 1b23c042e17d — Enhanced Weekend Adventure Planner 2026-10-09 08:00:38 → 08:17:46, failed (RuntimeError: Request timed out., notice delivered). No run since; next slot Fri 2026-10-16 08:00.
  • 0e753e9720fd — Facebook Birthday Monthly Scrape: no runs at all inside the retained window (monthly, next 2026-11-01), so its error/streak-0 row is still untested rather than fixed.

Sessions in the window: 18, all cron, no interactive session — a quiet day, in contrast to the previous window's talaria-ios research session. Busiest by message count: dca8482e4f76 (140), 1a8e90ea2271 stocktake (97), Wiki Audit (28), c0a9fa314542 (25). The 03:00 stocktake ran clean and pushed 0eee5f1 to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on disk — the known off-by-one), freshness audit 54 stale / 32 fresh / 7 no-date of 93 pages, Quartz step skipped on the documented path.

Mnemosyne (12:02 health run, green): working 760/786 embedded = 96.7%, episodic 1050/1056 = 99.4% — both above the 95% floor, so no reindex needed. Orphan embeddings 0. FTS5 aligned exactly (fts_working 786/786, fts_episodes 1056/1056, fts_facts 876/876). 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working / 1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to act on, and nothing to change on systems/mnemosyne beyond the coverage figures.

Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified). getMe against the telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 Unauthorized. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed, and there is no valid replacement anywhere on the box. The fetch path is healthy — GitHub trending pulled 100 repos and wrote 26 cache files in 24 h — so only digest.py's send() fails. Consequence: the Monday digest reached the DM only because the agent noticed the failure and re-ran with --dry-run and its own response was delivered. The 2026-10-05 "token-revocation claim refuted" entry was wrong and has been corrected: two green runs were read as proof the token worked, when a green cron row only means the agent finished. Fix: new token from @BotFather → config.json → re-run getMe. Credential present, value not recorded here.

Finding 2 — Camofox hit its memory ceiling during the Job Radar run. Mid-run probe of camofox-browser.service: Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M) with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom → /tabs and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and completed the extraction one tab at a time with immediate close. Same containment drop-in as documented (resource-containment.conf); the limits held, the workload is what needs bounding. → systems/browser-backend

Nothing added/removed: no new system, no retired job, so index.md is unchanged this cycle. No page archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs).

Still open after this cycle: GitHub Trending Radar token (new 2026-10-13); Camofox containment vs. multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; 3 non-clean last_status rows (0e753e9720fd, 1b23c042e17d, 066bb5191280); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.

Re-verified 2026-10-14

Window: 2026-10-13 00:00 → 23:59 AWST; live probes 2026-10-14 00:00–00:30 AWST. Fleet 29 jobs (26 enabled / 3 disabled) — unchanged this cycle. Open incidents 2 → 3.

Cron tally — the clean streak broke. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight at probe time, which finalises the day at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 → the first failed cron run since 2026-10-09 (a day that recorded 15).

Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent, alerted_at: NULL — failure streak 1, so under the 3-strike alert rule). This is the job's third failure in six days, all three on the opencode path: 10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode (1a8e90_789224ad589c), 10-09 timeout (1a8e90_dc6b2d40ec44), 10-13 404 no credentials (1a8e90_d11698bd7275). The first two resolved on 2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. each incident clears only on the next successful run of its own job. → incidents/2026-10-05-dead-model-pin-repin

Finding 2 — the failure stranded its own generated artifacts; this audit committed them. The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk, then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree: systems/skills-index.md (modified — date-only change: updated/verified_on 10-12 → 10-13; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked — 48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit. Structural point: the agent row is what commits, so a perfectly healthy script run still publishes nothing when the LLM turn dies. There is no green-row alternative — the row went red.

Retention floor moved. executions.db now holds 1,001 rows (1,000 terminal + 1 in flight); MAX_TERMINAL_EXECUTIONS = 1000 in cron/executions.py. Oldest retained terminal row: 2026-10-08 07:11. Days above the floor are complete; older ones undercount — 10-08 now reads 126 completed / 1 failed where its finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable.

Delivery. 10-13: 159 suppressed / 9 delivered / 1 no-record. Delivered = Daily Marketing Job Radar (dca8482e4f76), Mosquito Tracker (1b31187397a5 ×3), Tech & AI Newsletter Digest (85edf6772bca), Daily Birthday Checker (a432898b55a4), Gmail Action Digest (08eb86f02d37), FreshRSS Midday News Briefing (ecc1fc13655a), Evening Stand-Down Briefing (e530c89f2be0). No unexpected delivery; the two dead-thread delivery faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 10-01.

Mnemosyne. All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and no reindex. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the token markdown-bolded rather than bare; the delivery layer still suppressed it (delivery_outcome: suppressed), so the guard held — but the matcher is tolerant of wrapping rather than literal. → systems/mnemosyne

Non-clean last_status: 3 → 4. Cleared: fad5a86979d4 (success 10-12 03:05), dca8482e4f76 (success 10-13 07:02). Added: 1a8e90ea2271. Remaining: 0e753e9720fd (10-01), 1b23c042e17d (10-09), 066bb5191280 (10-01), 1a8e90ea2271 (10-13).

Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive session.

Pages touched: systems/scheduled-tasks.md (this block, frontmatter, non-clean list, retention note), systems/skills-index.md (committed stocktake artifact), current-state.md, index.md, log.md. Nothing added/removed: no new system, no retired job, no page archived → index.md structure unchanged.

Still open after this cycle: wiki-stocktake-maintenance opencode credential (new 2026-10-13 — watch the 10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off free-stack-opencode or repair OmniRoute's opencode provider credentials); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert; skills-index off-by-one; 4 non-clean last_status rows (above); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.

Re-verified 2026-10-15 / 2026-10-16 — the omni-101 connection-refused storm

Window: 2026-10-14 00:00 → 2026-10-16 00:00 AWST (two days — the 2026-10-14 23:59 audit died inside the same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x. Fleet 29 jobs (26 enabled / 3 disabled) — unchanged. Open incidents 3 → 12 (9 alerted / 3 detected).

The tally broke hard. 2026-10-14 = 160 completed / 12 failed; 2026-10-15 = 151 completed / 24 failed / 1 running. Against a four-day clean run (10-10 175/0, 10-11 172/0, 10-12 175/0, 10-13 167/1). 36 failure rows across 13 jobs — but only 24 are genuine model-call failures on 12 jobs; the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15).

Root cause. httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101 (config.yaml: custom provider name: omni-101). Surfaced as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. 328 Errno 111 lines in the live errors.log, 504 across rotations. Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → incidents/2026-10-14-omniroute-node-connection-refused-storm

Structural finding — no model-call failover anywhere in the fleet. Verified from jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable cron pin, custom:omni (CT600, omniroute:20128), sits on e530c89f2be0 — which is no_agent: true, a script-only job, so its provider field is vestigial. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni — that is the entire green of the fleet, and zero model calls completed in 34 hours. ⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is not: Stand-Down never calls a model. Never read a no_agent job's green row as provider-health evidence.

Worst-affected jobs: b63ed5817212 Mnemosyne Embedding Coverage ×6 (every slot for 30 h), 1b31187397a5 Mosquito Tracker ×4, ea79a2107e36 Mnemosyne Health Monitor ×3, 08eb86f02d37 Gmail Action Digest ×2, ecc1fc13655a FreshRSS Midday ×2, then one apiece for 85edf6772bca, dca8482e4f76, 0e519874d280, 06a0a1c8704f, c8d52904e1d6, 1a8e90ea2271, cbfe434a3b5e. Full table in the incident page. User-visible cost: Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went uncovered.

Recovery verified by schedule, not by probe alone. 1b31187397a5 completed 10-15 22:01:56 (streak 4 → 0); b63ed5817212 10-16 00:02:29; 96eb95ce12f2 10-16 00:01:36; ea79a2107e36 10-16 00:09:48. Live 10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required) and the same on omniroute:20128 / 100.88.81.19.

Non-clean last_status: 3 → 12. Cleared during the window: b63ed5817212, 1b31187397a5, ea79a2107e36 (all on successful post-storm runs). Added: the nine rows listed under Count above. Unchanged long-standing: 1b23c042e17d (10-09 error), 0e753e9720fd (10-01 error), 066bb5191280 (10-01 delivery_failed, dead Telegram thread_id 34795).

Mnemosyne. Both monitors sat on the failing node, so no coverage figures were refreshed in the window — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12. → systems/mnemosyne

Stocktake gap (self-healing). 1a8e90ea2271 failed 10-15 03:00 → raw/stocktake/freshness-20261015.txt was never written and systems/skills-index.md still carries the 10-14 stamp. Its next slot is 10-16 03:00, so the gap closes on its own — not regenerated by hand here, to avoid racing the scheduled run.

Sessions. 36 sessions in the two-day window, all cron, zero interactive — fourth consecutive window with no interactive session.

Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new), systems/omniroute.md, systems/mnemosyne.md, systems/scheduled-tasks.md (this block, frontmatter, Count), current-state.md, index.md, log.md. Nothing removed or archived: no retired job, no obsolete system → index.md structure gains one page.

Still open after this cycle: the no-failover topology itself (all 18 model-calling jobs on one OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads; Quartz rebuild step inert; skills-index off-by-one; 12 non-clean last_status rows; Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.