Files
agent-estate-wiki/systems/scheduled-tasks.md
T

17 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Scheduled Tasks system active 2026-07-22 2026-10-04 2026-10-04 high
cron
scheduled
automation
live-cron-list
live-jobs-json
cron-executions-db

Scheduled Tasks

Count

  • Total jobs: 30 — 27 enabled, 3 disabled (verified from live ~/.hermes/cron/jobs.json, 2026-10-03)
  • Failing (last run failed outright): 6 —
    • dca8482e4f76 — 403 MODEL_NOT_IN_PLAN, streak 7
    • 1a8e90ea2271 — HTTP 410, model retired upstream, streak 7
    • fad5a86979d4 — HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30)
    • 1b23c042e17d — new 2026-10-02 — HTTP 401, provider-connection auth expired (same pin as 06a0a1c8704f)
    • 06a0a1c8704f — new 2026-10-01 — HTTP 401, provider-connection auth expired
    • 0e753e9720fd — new 2026-10-01 — HTTP 400, same NVIDIA pin as fad5a86979d4
  • Failing (delivery only): 1 — 066bb5191280 (2026-10-01): the report itself generated; the send failed against the DM thread.
  • Resolved since the last entry: 1 — 1b23c042e17d's stale delivery_failed flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic 102 (delivery_outcome: delivered); it then failed on the model pin, above.
  • Shared model pin at risk: 3 enabled jobs run cliprox/terra — 06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5. Two of the three have now failed on that pin. The third (79f9bb41ecf5) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See incidents/2026-10-01-cliproxy-connection-auth-expired.
  • Streak recount (2026-10-03): recounted directly from ~/.hermes/cron/executions.db. dca8482e4f76 6 → 7 (one more daily run since), and 1a8e90ea2271 8 → 7 — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
  • Re-verified 2026-10-04 00:12 AWST — no change: still 30 jobs (27 enabled / 3 disabled) and the same 6 failing jobs; no new failure event landed inside the audit window. 79f9bb41ecf5 (Weekly Review, the last untested cliprox/terra job) drops at 2026-10-04 18:00 AWST — its result settles whether that pin is 3/3 broken. 066bb5191280 and 0e753e9720fd still carry the bare-telegram target warning (configured thread_id 34795 for telegram:1793951355 was not found; delivered without thread_id), confirmed live in hermes cron list.
  • Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)

Job Table

Name ID Schedule Delivery Skills Script Notes
AgentMail Rhino Inbox Poll 0a7f933d06e2 * * * * * origin terminal agentmail-poll.py disabled since 2026-07-27 (legacy)
AI Newsletter Curation for Patrick (Weekly) 06a0a1c8704f 0 10 * * 4 telegram:-1004321904721:346 — — ⚠ HTTP 401 ×1 (2026-10-01) — cliprox/terra provider connection auth expired; a second job on this pin failed 2026-10-02 (1b23c042e17d), so this is a confirmed credential fault, not a fluke. See incidents/2026-10-01-cliproxy-connection-auth-expired
Daily Birthday Checker a432898b55a4 30 7 * * * telegram:1793951355 — birthday_check.py
Daily Marketing Job Radar dca8482e4f76 0 7 * * 1-5 telegram:-1004321904721:1030 job-search-automation — ⚠ 403 MODEL_NOT_IN_PLAN ×7 — see Known Issues
DHL Tracking - Parcel 8052838382 62619aa6bcb3 0 8,11,14,17,20 * * * telegram:1793951355 — dhl_tracking_monitor.py no-agent; added 2026-09-24
Enhanced Weekend Adventure Planner 1b23c042e17d 0 8 * * 5 telegram:-1004321904721:102 — — 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — cliprox/terra connection auth expired, same pin as 06a0a1c8704f; no plan produced
Evening Stand-Down Briefing e530c89f2be0 45 18 * * 1-5 telegram:-1004321904721 — evening_standdown.py no-agent
Facebook Birthday Monthly Scrape 0e753e9720fd 0 9 1 * * telegram — — ⚠ HTTP 400 ×1 (2026-10-01) — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare telegram → home DM; retarget to a group topic
Facebook Birthday Weekly Registry Sync fad5a86979d4 0 3 * * 1 telegram — birthday_check.py ⚠ HTTP 400 ×3 — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; bare telegram target should move to a group topic
FreshRSS Midday News Briefing ecc1fc13655a 0 12 * * * telegram:-1004321904721:346 — freshrss_intelligence_v2.py
GitHub Trending Radar — Weekly Roundup c0a9fa314542 0 9 * * 1 telegram:1793951355 — — Token-revocation claim not reproduced at the last recorded run (2026-09-30 19:22, off-schedule: completed and delivered). The revoked-token (getMe → 401) diagnosis comes from an earlier run; the next scheduled run Mon 2026-10-05 09:00 confirms whether it still holds
Gmail Action Digest 08eb86f02d37 30 8 * * * telegram:-1004321904721:notifications — gmail_action_digest_v3.py
Important-mail monitor d40085da631c every 30m telegram:1793951355 — _cron_silent_monitor.py disabled since 2026-08-29 (Gmail read-state incident)
Job Tracker Dashboard Refresh 2fd50c87b0d0 0 */2 * * * telegram:-1004321904721:1030 — dashboard_watchdog.py no-agent
Mnemosyne Auto-Consolidate 0c771463c3cd every 30m local — consolidate_mnemosyne.sh no-agent
Mnemosyne DB Backup e5ed215fc91c 0 3 * * 0 telegram:1793951355 — mnemosyne-backup.sh no-agent
Mnemosyne Embedding Coverage Monitor b63ed5817212 0 */6 * * * telegram:-1004321904721:102 — check_embedding_coverage.py Alerts if coverage < 95%
Mnemosyne Health Monitor ea79a2107e36 0 */12 * * * telegram:-1004321904721:102 mnemosyne-operations — Reports only on exception
Mnemosyne Watchdog 96eb95ce12f2 */15 * * * * telegram:-1004321904721:102 — mnemosyne-watchdog.py no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30)
Mosquito Tracker 1b31187397a5 0 7,15,22 * * * telegram:-1004321904721:273 — mosquito_tracker.py Excludes 23:00–07:00
Perth New Restaurant & Cafe Openings (Weekly) c8d52904e1d6 0 18 * * 4 telegram:-1004321904721:102 — — delivery target corrected 2026-09-30; verified delivered, status ok
Political News Digest (Mon/Thu) 0e519874d280 0 8 * * 1,4 telegram:-1004321904721:346 — —
Rhino AgentMail Inbox Watcher f77b7e46bd09 every 5m origin — agentmail-rhino-watch.sh disabled since 2026-07-27 (legacy)
Tech & AI Newsletter Digest (Analytical) 85edf6772bca 0 7 * * * telegram:-1004321904721:346 — —
Uber Eats Monthly Spend Report 066bb5191280 0 2 1 * * telegram camofox-browser-automation, browser-backend-verification — ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic
Weekly Review 79f9bb41ecf5 0 18 * * 0 origin weekly-review — runs on cliprox/terra. Last run 2026-09-27 completed and delivered — before the connection lapsed. Next due 2026-10-04 18:00: the last untested job on this pin
weekly-save-restricted-bot-restart a688ed7b7ab0 0 3 * * 0 local — restart-save-restricted-bot.sh no-agent; script restored 2026-09-30
Wiki Audit — Nightly cbfe434a3b5e 59 23 * * * local task-approach — Reviews past 24h for wiki changes
wiki-stocktake-maintenance 1a8e90ea2271 0 3 * * * local — wiki-maintain.sh ⚠ HTTP 410 ×7 — qwen3.5:397b retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. The script itself is healthy — run by hand on 2026-10-03 and completed cleanly
🦝 Hermes Config Backup to Gitea (Daily) e27da6c34bbb 0 11 * * * origin — hermes-config-backup.sh no-agent

Removed Jobs

Name Removed Reason
Workday start reminder 2026-08-26 No longer present in live cron list
CT 221 Miner Watchdog 2026-08-26 Job d5516d0a5147 removed; CT221 status unknown/inspected
Review Agent Estate Phase One 2026-08-26 One-time job; completed 2026-08-08, now cleaned up

Disabled Jobs (not removed)

Name ID Disabled since Reason
Important-mail monitor d40085da631c 2026-08-29 Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state.
Rhino AgentMail Inbox Watcher f77b7e46bd09 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.
AgentMail Rhino Inbox Poll 0a7f933d06e2 2026-07-27 Legacy AgentMail polling; superseded by newer patterns.

Known Issues

Resolved 2026-09-30

  • Telegram thread delivery failures — root cause found and fixed: jobs fad5a86979d4, 1b23c042e17d, c8d52904e1d6 reported configured thread_id 34795 for telegram:1793951355 was not found / Message thread not found. The target was wrong, not the topic: a bare deliver: "telegram" resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). 1b23c042e17d and c8d52904e1d6 were retargeted to telegram:-1004321904721:102 and verified end-to-end. See systems/messaging-integrations.
  • Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause was ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2, run every 15 min by job 96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • weekly-save-restricted-bot-restart missing script (2026-08-30): Previously failed with "Script not found: restart-save-restricted-bot.sh". The script existed at /home/hermes/scripts/ but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now ok, streak 0.
  • Residual pre-fix dispatch errors cleared: cbfe434a3b5e returned to ok on its next run as predicted. The 1a8e90ea2271 error of the same era also cleared — its current failure is a different, upstream problem (below).
  • Stale delivery_failed flag on 1b23c042e17d cleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic 102.

Open

  • Daily Marketing Job Radar — 403 MODEL_NOT_IN_PLAN (2026-09-26 → 10-02, streak 7): Job dca8482e4f76 is pinned to custom:omni-101 / gpt-5.6-terra; the upstream (command-code) rejects it with GPT-5.6 Terra available in Pro and above plans. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See runbooks/job-radar.
  • cliprox/terra provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each): Jobs 06a0a1c8704f (2026-10-01) and 1b23c042e17d (2026-10-02) both failed with HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard (provider custom:Omniroute:20128, model cliprox/terra). Failure notices reached their topics; no output was produced. This is a credential/connection problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. 79f9bb41ecf5 is the remaining untested job on this pin (next run 2026-10-04 18:00 — re-check on 2026-10-05). Re-authenticate the OmniRoute connection before repinning anything. See incidents/2026-10-01-cliproxy-connection-auth-expired.
  • A delivered notice is not a successful job: both cliprox/terra failures recorded delivery_outcome: delivered because the failure notice is itself a deliverable. Read the execution status column, not the delivery outcome, when judging whether a job produced its output.
  • wiki-stocktake-maintenance — HTTP 410, model retired (streak 7): Job 1a8e90ea2271 is pinned to ollama-cloud / qwen3.5:397b, which Ollama retired on 2026-09-25 (qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT). It cannot self-recover — it needs a new model pin. The job's script is fine: wiki-maintain.sh was run by hand on 2026-10-03 and completed cleanly, regenerating systems/skills-index (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
  • Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1): Job fad5a86979d4 (weekly) is pinned to nvidia/deepseek-ai/deepseek-v4-flash; the provider reports the model is not available in the active live catalog for provider 'nvidia'. Confirmed as a shared-pin issue on 2026-10-01, when sibling job 0e753e9720fd (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (smart-route) instead of a raw model name.
  • GitHub Trending Radar — revoked Telegram bot token (claim now unverified): The recorded diagnosis is that job c0a9fa314542 builds its digest successfully, then fails the Telegram push (getMe → 401 Unauthorized), with the token in /home/hermes/workspace/github-trending-radar/config.json needing replacement from @BotFather. However its last recorded run (2026-09-30 19:22, off-schedule) completed and delivered, so the failure was not reproduced then. Treat "token revoked" as needing re-confirmation at the next scheduled run (Mon 2026-10-05 09:00) rather than as a standing fact. Digest content remains recoverable from the script's --dry-run output in the interim.
  • Stale provider key custom:omni (2 jobs): the provider was removed from custom_providers (the live 2026-10-01 set is litellm, guacano, Omniroute:20128, command-code, omni-101). Jobs e530c89f2be0 and d40085da631c still name it; both run smart-route, and the default custom provider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised to custom:omni-101.
  • Bare telegram delivery targets (3 jobs) — demonstrably breaking sends: 0e753e9720fd, fad5a86979d4, 066bb5191280 deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: 066bb5191280 (02:11) and 0e753e9720fd (09:01) both recorded delivery_outcome: failed with configured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three to telegram:-1004321904721:<topic>. See systems/messaging-integrations.
  • Two wiki trees, drifted in both directions (NEW 2026-10-02): /home/hermes/workspace/agent-estate-wiki is the canonical source (Gitea remote Anthony/agent-estate-wiki, Quartz build, wiki-maintain.sh target). /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads — pages nested under wiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md, systems/nextcloud.md, three incidents/ pages exist only in the vault; raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md only in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent.
  • Work mailbox unreachable (non-cron; affects on-demand lookups): the work Outlook account is configured with password auth while the tenant advertises only XOAuth2, so himalaya -a work fails with login not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.

Historical

  • Mnemosyne Watchdog incident (2026-08-19): Job 96eb95ce12f2 cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne.
  • Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
  • Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to smart-route or gpt-5.6-terra.
  • Wiki Audit model drift (2026-08-07): Global inference config change caused RuntimeError: Skipped to prevent unintended spend on an unpinned job.