20 KiB
20 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduled Tasks | system | active | 2026-07-22 | 2026-10-07 | 2026-10-07 | high |
|
|
Scheduled Tasks
Count
- Total jobs: 30 — 27 enabled, 3 disabled (verified from live
~/.hermes/cron/jobs.json, 2026-10-06) - Failing (last run failed outright): 0 — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider
custom:omni-101using routing aliases, and everyfailure_streakinjobs.jsonnow reads 0. Four of the seven jobs had still carried a pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd,1a8e90ea2271) — a stale field, not a live failure; it clears on each job's next scheduled run.1a8e90ea2271cleared on 2026-10-06 03:00, running on schedule for the first time since 2026-09-25; the remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are due Fri 08:00 / Mon 03:00 / 1st 09:00. See incidents/2026-10-05-dead-model-pin-repin.- Historical (2026-10-01 → 2026-10-05), now all remediated:
dca8482e4f76403MODEL_NOT_IN_PLAN×7;1a8e90ea2271HTTP 410 ×11 (retiredqwen3.5:397b);fad5a86979d4HTTP 400 ×3;0e753e9720fdHTTP 400 ×1;06a0a1c8704f,1b23c042e17dand79f9bb41ecf5HTTP 401 (thecliprox/terraconnection auth expiry, settled 3/3 on 2026-10-04).
- Historical (2026-10-01 → 2026-10-05), now all remediated:
- Failing (delivery only): 1 —
066bb5191280: the report itself generated; the send failed against the DM thread (baretelegramtarget, 2026-10-01). Unchanged this window. - Repin applied 2026-10-05:
free-stack×6 (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271,fad5a86979d4,0e753e9720fd) andsmart-route×1 (dca8482e4f76), all oncustom:omni-101. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it.free-stackis in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - Verified live, not assumed:
79f9bb41ecf5succeeded on a manual run 2026-10-05 23:39 and06a0a1c8704fat 23:45 (both incidents auto-resolved);dca8482e4f76completed 23:56 and delivered totelegram:-1004321904721:1030(incident resolved 2026-10-06 00:05:07); and1a8e90ea2271completed automatically on 2026-10-06 03:00. Four of the seven are therefore verified by live run. The remaining three (1b23c042e17d,fad5a86979d4,0e753e9720fd) are repinned but not yet re-run — they are due on their normal schedules. - Streak counting note: the
N failures in a rowfigure inhermes cron listcomes from thefailure_streakcounter injobs.json.executions.dbretains only rows from 2026-09-30 onward, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: all 0. - Re-verified 2026-10-06 AWST: job count unchanged at 30 (27 enabled / 3 disabled).
066bb5191280and0e753e9720fdstill carry the bare-telegramtarget warning (configured thread_id 34795 for telegram:1793951355 was not found). - Re-verified 2026-10-07 AWST: job count unchanged at 30 (27 enabled / 3 disabled); 0 failing — all 30
failure_streakcounters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 180completed, 0failed, 1running(the audit itself); no interactive (non-cron) sessions. The stocktake job1a8e90ea2271ran on schedule at 03:00 and regenerated systems/skills-index (344 → 346 skills) plusraw/stocktake/freshness-20261006.txt, committing to Gitea as33f4e95— the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repinlast_status: error(1b23c042e17d,fad5a86979d4,0e753e9720fd). - A stale notice, not a live outage, drove this review. The interactive session on 2026-10-05 23:16 began because Anthony replied to a
85edf6772bcafailure notice that had been sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_at(and thecron_incidentsstate) before acting on a notice. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, 0 failing (2026-10-06) → 30, 0 failing (2026-10-07)
Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
* * * * * |
origin | terminal | agentmail-poll.py |
disabled since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | 06a0a1c8704f |
0 10 * * 4 |
telegram:-1004321904721:346 | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was custom:Omniroute:20128 / cliprox/terra, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live last_status: ok, streak 0 |
| Daily Birthday Checker | a432898b55a4 |
30 7 * * * |
telegram:1793951355 | — | birthday_check.py |
|
| Daily Marketing Job Radar | dca8482e4f76 |
0 7 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ✅ Repinned 2026-10-05 → custom:omni-101 / smart-route (was command-code / gpt-5.6-terra, 403 MODEL_NOT_IN_PLAN ×7). A manual run completed at 23:56 and delivered to topic 1030 — first successful run since 2026-09-25. Live last_status: ok, streak 0 |
| DHL Tracking - Parcel 8052838382 | 62619aa6bcb3 |
0 8,11,14,17,20 * * * |
telegram:1793951355 | — | dhl_tracking_monitor.py |
no-agent; added 2026-09-24 |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
0 8 * * 5 |
telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix confirmed working (delivered 2026-10-02, flag cleared). ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the failed cliprox/terra pin, HTTP 401 ×1). Not yet re-run — carries a stale last_status: error until Fri 08:00; streak 0 |
| Evening Stand-Down Briefing | e530c89f2be0 |
45 18 * * 1-5 |
telegram:-1004321904721 | — | evening_standdown.py |
no-agent |
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
0 9 1 * * |
telegram | — | — | ✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×1). Streak 0; stale last_status: error until the 1st. ⚠ Bare telegram still targets the home DM — retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
0 3 * * 1 |
telegram | — | birthday_check.py |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the catalog-missing nvidia/deepseek-ai/deepseek-v4-flash pin, HTTP 400 ×3). Streak 0; stale last_status: error until Mon 03:00. ⚠ Bare telegram target should move to a group topic |
| FreshRSS Midday News Briefing | ecc1fc13655a |
0 12 * * * |
telegram:-1004321904721:346 | — | freshrss_intelligence_v2.py |
|
| GitHub Trending Radar — Weekly Roundup | c0a9fa314542 |
0 9 * * 1 |
telegram:1793951355 | — | — | ✅ Token-revocation claim refuted. The scheduled run on 2026-10-05 09:00 completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run — so the getMe → 401 diagnosis no longer applies and no token replacement is needed. Still delivers to the bare home DM, which works |
| Gmail Action Digest | 08eb86f02d37 |
30 8 * * * |
telegram:-1004321904721:notifications | — | gmail_action_digest_v3.py |
|
| Important-mail monitor | d40085da631c |
every 30m |
telegram:1793951355 | — | _cron_silent_monitor.py |
disabled since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | 2fd50c87b0d0 |
0 */2 * * * |
telegram:-1004321904721:1030 | — | dashboard_watchdog.py |
no-agent |
| Mnemosyne Auto-Consolidate | 0c771463c3cd |
every 30m |
local | — | consolidate_mnemosyne.sh |
no-agent |
| Mnemosyne DB Backup | e5ed215fc91c |
0 3 * * 0 |
telegram:1793951355 | — | mnemosyne-backup.sh |
no-agent |
| Mnemosyne Embedding Coverage Monitor | b63ed5817212 |
0 */6 * * * |
telegram:-1004321904721:102 | — | check_embedding_coverage.py |
Alerts if coverage < 95% |
| Mnemosyne Health Monitor | ea79a2107e36 |
0 */12 * * * |
telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Watchdog | 96eb95ce12f2 |
*/15 * * * * |
telegram:-1004321904721:102 | — | mnemosyne-watchdog.py |
no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30) |
| Mosquito Tracker | 1b31187397a5 |
0 7,15,22 * * * |
telegram:-1004321904721:273 | — | mosquito_tracker.py |
Excludes 23:00–07:00 |
| Perth New Restaurant & Cafe Openings (Weekly) | c8d52904e1d6 |
0 18 * * 4 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified delivered, status ok |
| Political News Digest (Mon/Thu) | 0e519874d280 |
0 8 * * 1,4 |
telegram:-1004321904721:346 | — | — | |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
every 5m |
origin | — | agentmail-rhino-watch.sh |
disabled since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | 85edf6772bca |
0 7 * * * |
telegram:-1004321904721:346 | — | — | |
| Uber Eats Monthly Spend Report | 066bb5191280 |
0 2 1 * * |
telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic |
| Weekly Review | 79f9bb41ecf5 |
0 18 * * 0 |
origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the cliprox/terra pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ Repinned 2026-10-05 23:39 → custom:omni-101 / free-stack and verified by a live manual run; the incident auto-resolved. Now last_status: ok, streak 0 |
| weekly-save-restricted-bot-restart | a688ed7b7ab0 |
0 3 * * 0 |
local | — | restart-save-restricted-bot.sh |
no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | cbfe434a3b5e |
59 23 * * * |
local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
0 3 * * * |
local | — | wiki-maintain.sh |
✅ Repinned 2026-10-05 → custom:omni-101 / free-stack (was the retired qwen3.5:397b, HTTP 410 ×11). ✅ Verified by live run 2026-10-06 03:00 — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as 33f4e95. Live last_status: ok, streak 0. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea — the script itself was never broken |
| 🦝 Hermes Config Backup to Gitea (Daily) | e27da6c34bbb |
0 11 * * * |
origin | — | hermes-config-backup.sh |
no-agent |
Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job d5516d0a5147 removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | d40085da631c |
2026-08-29 | Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state. |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
Known Issues
Resolved 2026-10-05
- Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05): Seven jobs were each pinned to a raw model name that had died upstream —
cliprox/terra(HTTP 401, connection auth expired, 3 jobs),ollama-cloud/qwen3.5:397b(HTTP 410, retired 2026-09-25),nvidia/deepseek-ai/deepseek-v4-flash(HTTP 400, dropped from the live catalog, 2 jobs) andcommand-code/gpt-5.6-terra(HTTP 403MODEL_NOT_IN_PLAN) — across 2026-10-01 → 2026-10-05. All seven were repinned to providercustom:omni-101using routing aliases —free-stack×6,smart-route×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it.failure_streakis now 0 on all 30 jobs, down from 7 failing. Three were verified by live run:79f9bb41ecf5(23:39),06a0a1c8704f(23:45) anddca8482e4f76(23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stalelast_status: errorfield clears on each one's next scheduled run. See incidents/2026-10-05-dead-model-pin-repin.- The
cliprox/terraincident is closed for cron purposes: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See incidents/2026-10-01-cliproxy-connection-auth-expired. wiki-stocktake-maintenanceresumes its 03:00 schedule, sowiki-maintain.shno longer needs a daily manual run.
- The
- GitHub Trending Radar
c0a9fa314542— token-revocation claim refuted (2026-10-05): the scheduled Mon 09:00 run completed cleanly — its second consecutive success after the off-schedule 2026-09-30 19:22 run. ThegetMe→ 401 revoked-token diagnosis no longer applies and no token replacement is needed. It still delivers to the bare home DM, which works. - A stale failure notice, not a live outage, opened the 2026-10-05 review: the interactive session began because Anthony acted on a
85edf6772bcafailure notice sitting in topic346since the 2026-09-29 07:00 run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were alreadyresolvedon 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice.
Resolved 2026-09-30
- Telegram thread delivery failures — root cause found and fixed: jobs
fad5a86979d4,1b23c042e17d,c8d52904e1d6reportedconfigured thread_id 34795 for telegram:1793951355 was not found/Message thread not found. The target was wrong, not the topic: a baredeliver: "telegram"resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong).1b23c042e17dandc8d52904e1d6were retargeted totelegram:-1004321904721:102and verified end-to-end. See systems/messaging-integrations. - Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause wasModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2, run every 15 min by job96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps. weekly-save-restricted-bot-restartmissing script (2026-08-30): Previously failed with "Script not found:restart-save-restricted-bot.sh". The script existed at/home/hermes/scripts/but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job nowok, streak 0.- Residual pre-fix dispatch errors cleared:
cbfe434a3b5ereturned tookon its next run as predicted. The1a8e90ea2271error of the same era also cleared — its current failure is a different, upstream problem (below). - Stale
delivery_failedflag on1b23c042e17dcleared (confirmed 2026-10-02): the 2026-09-30 retarget held — the job's next run delivered to topic102.
Open
- A delivered notice is not a successful job:
cliprox/terrafailures recordeddelivery_outcome: deliveredbecause the failure notice is itself a deliverable. Read the executionstatuscolumn, not the delivery outcome, when judging whether a job produced its output. - Stale provider key
custom:omni(1 enabled + 1 disabled): the provider was removed fromcustom_providers(live set on 2026-10-01:litellm,guacano,Omniroute:20128,command-code,omni-101). Re-verified 2026-10-06:e530c89f2be0(enabled) andd40085da631c(disabled) still name it; both runsmart-route, and the defaultcustomprovider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised tocustom:omni-101. The 2026-10-05 repin moved seven jobs to an explicitcustom:omni-101, leaving these two as the only remainingcustom:omnireferences. - Bare
telegramdelivery targets (3 jobs) — demonstrably breaking sends:0e753e9720fd,fad5a86979d4,066bb5191280deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical:066bb5191280(02:11) and0e753e9720fd(09:01) both recordeddelivery_outcome: failedwithconfigured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three totelegram:-1004321904721:<topic>. See systems/messaging-integrations. - Two wiki trees, drifted in both directions (NEW 2026-10-02):
/home/hermes/workspace/agent-estate-wikiis the canonical source (Gitea remoteAnthony/agent-estate-wiki, Quartz build,wiki-maintain.shtarget)./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads — pages nested underwiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md,systems/nextcloud.md, threeincidents/pages exist only in the vault;raw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdonly in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent. - Work mailbox unreachable (non-cron; affects on-demand lookups): the
workOutlook account is configured with password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workfails withlogin not supported. No cron job polls it. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
Historical
- Mnemosyne Watchdog incident (2026-08-19): Job
96eb95ce12f2cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne. - Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
- Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to
smart-routeorgpt-5.6-terra. - Wiki Audit model drift (2026-08-07): Global inference config change caused
RuntimeError: Skipped to prevent unintended spendon an unpinned job.