18 KiB
18 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduled Tasks | system | active | 2026-07-22 | 2026-10-05 | 2026-10-05 | high |
|
|
Scheduled Tasks
Count
- Total jobs: 30 — 27 enabled, 3 disabled (verified from live
~/.hermes/cron/jobs.json, 2026-10-05) - Failing (last run failed outright): 7 —
dca8482e4f76— 403MODEL_NOT_IN_PLAN, streak 71a8e90ea2271— HTTP 410, model retired upstream, streak 11fad5a86979d4— HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30)79f9bb41ecf5— new 2026-10-04 — HTTP 401, provider-connection auth expired; the third and final job on thecliprox/terrapin (see below)1b23c042e17d— new 2026-10-02 — HTTP 401, provider-connection auth expired (same pin as06a0a1c8704f)06a0a1c8704f— new 2026-10-01 — HTTP 401, provider-connection auth expired0e753e9720fd— new 2026-10-01 — HTTP 400, same NVIDIA pin asfad5a86979d4
- Failing (delivery only): 1 —
066bb5191280(2026-10-01): the report itself generated; the send failed against the DM thread. - Resolved since the last entry: 1 —
1b23c042e17d's staledelivery_failedflag cleared exactly as predicted. Its 2026-10-02 run delivered to topic102(delivery_outcome: delivered); it then failed on the model pin, above. cliprox/terrapin — settled 2026-10-04: 3 of 3 broken. All three enabled jobs on that pin (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5) have now failed with the identicalHTTP 401: All 1 connection(s) authentication expired.79f9bb41ecf5(Weekly Review) ran 2026-10-04 18:00 and failed as the last untested job — settling it as a provider-connection credential fault, not a per-job fluke. See incidents/2026-10-01-cliproxy-connection-auth-expired.- Streak counting note: the
N failures in a rowfigure inhermes cron listcomes from thefailure_streakcounter injobs.json.executions.dbnow retains only rows from 2026-09-30 onward (1,001 rows total), so DB-based recounts undercount older streaks. Live counters as of 2026-10-05:1a8e90ea2271×11,dca8482e4f76×7,fad5a86979d4×3. - Re-verified 2026-10-05 00:01 AWST: job count unchanged at 30 (27 enabled / 3 disabled). Failing set moved 6 → 7 — the only new failure event in the window is
79f9bb41ecf5's 2026-10-04 18:00 run. No interactive (non-cron) sessions occurred in the past 24h, so the only durable changes are cron-fleet.066bb5191280and0e753e9720fdstill carry the bare-telegramtarget warning (configured thread_id 34795 for telegram:1793951355 was not found), confirmed live inhermes cron list. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05)
Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
* * * * * |
origin | terminal | agentmail-poll.py |
disabled since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | 06a0a1c8704f |
0 10 * * 4 |
telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — cliprox/terra provider connection auth expired. All 3 jobs on this pin have now failed on it (see 1b23c042e17d, 79f9bb41ecf5), so this is a confirmed credential fault. See incidents/2026-10-01-cliproxy-connection-auth-expired |
| Daily Birthday Checker | a432898b55a4 |
30 7 * * * |
telegram:1793951355 | — | birthday_check.py |
|
| Daily Marketing Job Radar | dca8482e4f76 |
0 7 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 MODEL_NOT_IN_PLAN ×7 — see Known Issues |
| DHL Tracking - Parcel 8052838382 | 62619aa6bcb3 |
0 8,11,14,17,20 * * * |
telegram:1793951355 | — | dhl_tracking_monitor.py |
no-agent; added 2026-09-24 |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
0 8 * * 5 |
telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — cliprox/terra connection auth expired, same pin as 06a0a1c8704f and 79f9bb41ecf5 (pin now 3/3 broken); no plan produced |
| Evening Stand-Down Briefing | e530c89f2be0 |
45 18 * * 1-5 |
telegram:-1004321904721 | — | evening_standdown.py |
no-agent |
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
0 9 1 * * |
telegram | — | — | ⚠ HTTP 400 ×1 (2026-10-01) — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare telegram → home DM; retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
0 3 * * 1 |
telegram | — | birthday_check.py |
⚠ HTTP 400 ×3 — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; bare telegram target should move to a group topic |
| FreshRSS Midday News Briefing | ecc1fc13655a |
0 12 * * * |
telegram:-1004321904721:346 | — | freshrss_intelligence_v2.py |
|
| GitHub Trending Radar — Weekly Roundup | c0a9fa314542 |
0 9 * * 1 |
telegram:1793951355 | — | — | Token-revocation claim not reproduced at the last recorded run (2026-09-30 19:22, off-schedule: completed and delivered). The revoked-token (getMe → 401) diagnosis comes from an earlier run; the next scheduled run Mon 2026-10-05 09:00 confirms whether it still holds |
| Gmail Action Digest | 08eb86f02d37 |
30 8 * * * |
telegram:-1004321904721:notifications | — | gmail_action_digest_v3.py |
|
| Important-mail monitor | d40085da631c |
every 30m |
telegram:1793951355 | — | _cron_silent_monitor.py |
disabled since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | 2fd50c87b0d0 |
0 */2 * * * |
telegram:-1004321904721:1030 | — | dashboard_watchdog.py |
no-agent |
| Mnemosyne Auto-Consolidate | 0c771463c3cd |
every 30m |
local | — | consolidate_mnemosyne.sh |
no-agent |
| Mnemosyne DB Backup | e5ed215fc91c |
0 3 * * 0 |
telegram:1793951355 | — | mnemosyne-backup.sh |
no-agent |
| Mnemosyne Embedding Coverage Monitor | b63ed5817212 |
0 */6 * * * |
telegram:-1004321904721:102 | — | check_embedding_coverage.py |
Alerts if coverage < 95% |
| Mnemosyne Health Monitor | ea79a2107e36 |
0 */12 * * * |
telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Watchdog | 96eb95ce12f2 |
*/15 * * * * |
telegram:-1004321904721:102 | — | mnemosyne-watchdog.py |
no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30) |
| Mosquito Tracker | 1b31187397a5 |
0 7,15,22 * * * |
telegram:-1004321904721:273 | — | mosquito_tracker.py |
Excludes 23:00–07:00 |
| Perth New Restaurant & Cafe Openings (Weekly) | c8d52904e1d6 |
0 18 * * 4 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified delivered, status ok |
| Political News Digest (Mon/Thu) | 0e519874d280 |
0 8 * * 1,4 |
telegram:-1004321904721:346 | — | — | |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
every 5m |
origin | — | agentmail-rhino-watch.sh |
disabled since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | 85edf6772bca |
0 7 * * * |
telegram:-1004321904721:346 | — | — | |
| Uber Eats Monthly Spend Report | 066bb5191280 |
0 2 1 * * |
telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic |
| Weekly Review | 79f9bb41ecf5 |
0 18 * * 0 |
origin | weekly-review | — | runs on cliprox/terra. ⚠ failed 2026-10-04 18:00 with the pin's HTTP 401 (All 1 connection(s) authentication expired) — the last untested job on that pin, now confirming it 3/3 broken. No review produced. See incidents/2026-10-01-cliproxy-connection-auth-expired |
| weekly-save-restricted-bot-restart | a688ed7b7ab0 |
0 3 * * 0 |
local | — | restart-save-restricted-bot.sh |
no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | cbfe434a3b5e |
59 23 * * * |
local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
0 3 * * * |
local | — | wiki-maintain.sh |
⚠ HTTP 410 ×11 — qwen3.5:397b retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. The script itself is healthy — run by hand on 2026-10-03 and completed cleanly |
| 🦝 Hermes Config Backup to Gitea (Daily) | e27da6c34bbb |
0 11 * * * |
origin | — | hermes-config-backup.sh |
no-agent |
Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job d5516d0a5147 removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | d40085da631c |
2026-08-29 | Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state. |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
Known Issues
Resolved 2026-09-30
- Telegram thread delivery failures — root cause found and fixed: jobs
fad5a86979d4,1b23c042e17d,c8d52904e1d6reportedconfigured thread_id 34795 for telegram:1793951355 was not found/Message thread not found. The target was wrong, not the topic: a baredeliver: "telegram"resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong).1b23c042e17dandc8d52904e1d6were retargeted totelegram:-1004321904721:102and verified end-to-end. See systems/messaging-integrations. - Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause wasModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2, run every 15 min by job96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps. weekly-save-restricted-bot-restartmissing script (2026-08-30): Previously failed with "Script not found:restart-save-restricted-bot.sh". The script existed at/home/hermes/scripts/but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job nowok, streak 0.- Residual pre-fix dispatch errors cleared:
cbfe434a3b5ereturned tookon its next run as predicted. The1a8e90ea2271error of the same era also cleared — its current failure is a different, upstream problem (below). - Stale
delivery_failedflag on1b23c042e17dcleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic102.
Open
- Daily Marketing Job Radar — 403
MODEL_NOT_IN_PLAN(2026-09-26 → 10-02, streak 7): Jobdca8482e4f76is pinned tocustom:omni-101/gpt-5.6-terra; the upstream (command-code) rejects it withGPT-5.6 Terra available in Pro and above plans. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See runbooks/job-radar. cliprox/terraprovider connection auth expired — 3 of 3 jobs confirmed (2026-10-04): Jobs06a0a1c8704f(2026-10-01),1b23c042e17d(2026-10-02) and79f9bb41ecf5(2026-10-04) all failed withHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard(providercustom:Omniroute:20128, modelcliprox/terra). Failure notices reached their topics; no output was produced. Every enabled job on this pin has now failed on it — a credential/connection problem at the proxy, not a missing model, and it will not self-recover until the connection is re-authenticated. Re-authenticate the OmniRoute connection before repinning anything. See incidents/2026-10-01-cliproxy-connection-auth-expired.- A delivered notice is not a successful job: all three
cliprox/terrafailures recordeddelivery_outcome: deliveredbecause the failure notice is itself a deliverable. Read the executionstatuscolumn, not the delivery outcome, when judging whether a job produced its output. - wiki-stocktake-maintenance — HTTP 410, model retired (streak 11): Job
1a8e90ea2271is pinned toollama-cloud / qwen3.5:397b, which Ollama retired on 2026-09-25 (qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT). It cannot self-recover — it needs a new model pin. The job's script is fine:wiki-maintain.shwas run by hand on 2026-10-03 and completed cleanly, regenerating systems/skills-index (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned. It failed again on 2026-10-04 03:00 (same 410). - Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1): Job
fad5a86979d4(weekly) is pinned tonvidia/deepseek-ai/deepseek-v4-flash; the provider reports the model isnot available in the active live catalog for provider 'nvidia'. Confirmed as a shared-pin issue on 2026-10-01, when sibling job0e753e9720fd(monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (smart-route) instead of a raw model name. - GitHub Trending Radar — revoked Telegram bot token (claim now unverified): The recorded diagnosis is that job
c0a9fa314542builds its digest successfully, then fails the Telegram push (getMe→ 401 Unauthorized), with the token in/home/hermes/workspace/github-trending-radar/config.jsonneeding replacement from @BotFather. However its last recorded run (2026-09-30 19:22, off-schedule) completed and delivered, so the failure was not reproduced then. Treat "token revoked" as needing re-confirmation at the next scheduled run (Mon 2026-10-05 09:00) rather than as a standing fact. Digest content remains recoverable from the script's--dry-runoutput in the interim. - Stale provider key
custom:omni(2 jobs): the provider was removed fromcustom_providers(the live 2026-10-01 set islitellm,guacano,Omniroute:20128,command-code,omni-101). Jobse530c89f2be0andd40085da631cstill name it; both runsmart-route, and the defaultcustomprovider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised tocustom:omni-101. - Bare
telegramdelivery targets (3 jobs) — demonstrably breaking sends:0e753e9720fd,fad5a86979d4,066bb5191280deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical:066bb5191280(02:11) and0e753e9720fd(09:01) both recordeddelivery_outcome: failedwithconfigured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three totelegram:-1004321904721:<topic>. See systems/messaging-integrations. - Two wiki trees, drifted in both directions (NEW 2026-10-02):
/home/hermes/workspace/agent-estate-wikiis the canonical source (Gitea remoteAnthony/agent-estate-wiki, Quartz build,wiki-maintain.shtarget)./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads — pages nested underwiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md,systems/nextcloud.md, threeincidents/pages exist only in the vault;raw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdonly in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent. - Work mailbox unreachable (non-cron; affects on-demand lookups): the
workOutlook account is configured with password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workfails withlogin not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
Historical
- Mnemosyne Watchdog incident (2026-08-19): Job
96eb95ce12f2cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne. - Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
- Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to
smart-routeorgpt-5.6-terra. - Wiki Audit model drift (2026-08-07): Global inference config change caused
RuntimeError: Skipped to prevent unintended spendon an unpinned job.