Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh
This commit is contained in:
+66
-41
@@ -3,64 +3,89 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-09-19
|
||||
verified_on: 2026-09-19
|
||||
updated: 2026-09-30
|
||||
verified_on: 2026-09-30
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list]
|
||||
sources: [live-cron-list, live-jobs-json]
|
||||
---
|
||||
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Active jobs:** 26 (verified from live `hermes cron list`, 2026-09-19)
|
||||
- **Previously 29** — 3 jobs removed or consolidated since 2026-08-16.
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-09-30)
|
||||
- **Failing:** 4 (`dca8482e4f76` model-plan 403 ×5; `fad5a86979d4` thread; `cbfe434a3b5e` + `1a8e90ea2271` residual pre-fix dispatch error)
|
||||
- **Delivery-only failures:** 2 (`c8d52904e1d6`, `1b23c042e17d` — content ran, Telegram thread missing)
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30)
|
||||
|
||||
## Job Table
|
||||
|
||||
| Name | Schedule | Delivery | Skills | Script | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Facebook Birthday Monthly Scrape | `0 9 1 * *` | telegram | — | — | ⚠ thread_id not found |
|
||||
| Facebook Birthday Weekly Registry Sync | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ thread_id not found |
|
||||
| Daily Birthday Checker | `30 7 * * *` | telegram | — | `birthday_check.py` | no-agent |
|
||||
| Daily Marketing Job Radar | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | Primary `gpt-5.6-terra`; see [[runbooks/job-radar]] |
|
||||
| Perth New Restaurant & Cafe Openings (Weekly) | `0 18 * * 4` | telegram | — | — | |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
|
||||
| Enhanced Weekend Adventure Planner | `0 8 * * 5` | telegram | — | — | |
|
||||
| Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||
| Political News Digest (Mon/Thu) | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||
| Uber Eats Monthly Spend Report | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
||||
| Gmail Action Digest | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||
| FreshRSS Midday News Briefing | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||
| Workday start reminder | `0 9 * * 1-5` | telegram | — | — | |
|
||||
| Important-mail monitor | `every 30m` | telegram:1793951355 | — | — | Silent when no important mail |
|
||||
| Mnemosyne Auto-Consolidate | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||
| Mnemosyne DB Backup | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||
| Mnemosyne Health Monitor | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||
| Mnemosyne Watchdog | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts |
|
||||
| Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
||||
| CT 221 Miner Watchdog | `every 15m` | telegram:1793951355 | pve-lxc-intrusion-detection | `ct221-miner-watchdog.sh` | Added 2026-08-07 after crypto miner incident |
|
||||
| Mosquito Tracker | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||
| Weekly Review | `0 18 * * 0` | origin | weekly-review | — | |
|
||||
| Evening Stand-Down Briefing | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Job Tracker Dashboard Refresh | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||
| Wiki Audit — Nightly | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||
| GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||
| Review Agent Estate Phase One | once at 2026-08-08 18:33 | origin | — | — | ✅ Completed 2026-08-08. See [[current-state]]. |
|
||||
| wiki-stocktake-maintenance | `0 3 * * *` | local | — | `wiki-maintain.sh` | no-agent |
|
||||
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
|
||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×5 — see Known Issues |
|
||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram | — | — | ⚠ `thread_id 34795` not found → delivered without thread |
|
||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ `thread_id 34795` not found → delivered without thread |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
||||
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram | — | — | ⚠ delivery error "Message thread not found" |
|
||||
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | |
|
||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||
|
||||
## Removed Jobs
|
||||
| Name | Removed | Reason |
|
||||
|---|---|---|
|
||||
| GitHub Trending Radar — Daily Digest | 2026-08-02 | Stopped by user. Weekly roundup still active. |
|
||||
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
||||
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
||||
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
||||
|
||||
## Disabled Jobs (not removed)
|
||||
| Name | ID | Disabled since | Reason |
|
||||
|---|---|---|---|
|
||||
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||
|
||||
## Known Issues
|
||||
- **Session storage failures (2026-08-10):** Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError ("session storage could not be written"). Disk at 83% (11GB free), `state.db` at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See [[current-state]] for details.
|
||||
- **Wiki Audit cron model drift (2026-08-07):** Global inference config changed (`smart-route` → `deepseek-paid-smart`) and the job is unpinned, causing `RuntimeError: Skipped to prevent unintended spend`. Needs pinning: `cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>`.
|
||||
- **Facebook Birthday thread_id warnings:** Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread.
|
||||
- **Model repinning (2026-08-08):** 13 cron jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. Dead combos included `freeapi/minimax-m2.7`, `freeapi/auto`, `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`. Jobs now running on valid model pins.
|
||||
|
||||
### Resolved 2026-09-30
|
||||
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. 10/10 managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". Script restored 2026-09-30 19:17; job now `ok`, streak 0.
|
||||
|
||||
### Open
|
||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30, streak 5):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change. See [[runbooks/job-radar]].
|
||||
- **Telegram thread delivery failures (`telegram:1793951355`):** `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`; `c8d52904e1d6` reports `Message thread not found` (last_status `delivery_failed`). Content runs successfully — only the thread target is wrong. Fix: either recreate the topic or clear `thread_id` from the jobs.
|
||||
- **Residual dispatch errors:** `cbfe434a3b5e` (streak 5) and `1a8e90ea2271` (streak 6) still show the pre-2026-09-30 worker error from runs made before the dependency guard was installed (23:59 AWST and 03:00 AWST respectively). Expected to clear on their next runs — re-check 2026-10-01 03:00.
|
||||
|
||||
### Historical
|
||||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
||||
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
|
||||
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
|
||||
|
||||
## Related
|
||||
- [[systems/mnemosyne]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||
|
||||
Reference in New Issue
Block a user