- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
14 KiB
14 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scheduled Tasks | system | active | 2026-07-22 | 2026-10-02 | 2026-10-02 | high |
|
|
Scheduled Tasks
Count
- Total jobs: 30 — 27 enabled, 3 disabled (verified from live
~/.hermes/cron/jobs.json, 2026-10-02) - Failing (last run): 6 —
dca8482e4f76— 403MODEL_NOT_IN_PLAN, streak 61a8e90ea2271— HTTP 410, model retired upstream, streak 8fad5a86979d4— HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30)06a0a1c8704f— new 2026-10-01 — HTTP 401, provider-connection auth expired0e753e9720fd— new 2026-10-01 — HTTP 400, same NVIDIA pin asfad5a86979d4066bb5191280— new 2026-10-01 — delivery failed (DM thread); the report itself generated
- Stale status flag: 1 —
1b23c042e17dstill readsdelivery_failedfrom its pre-fix run. Its delivery target was corrected on 2026-09-30; the next run (2026-10-02 08:00) clears the flag — but that run is also exposed to thecliprox/terraauth fault below. - Shared model pin at risk: 3 enabled jobs run
cliprox/terra—06a0a1c8704f,1b23c042e17d,79f9bb41ecf5. One of the three (06a0a1c8704f) has already failed on that pin. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02)
Job Table
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---|---|
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
* * * * * |
origin | terminal | agentmail-poll.py |
disabled since 2026-07-27 (legacy) |
| AI Newsletter Curation for Patrick (Weekly) | 06a0a1c8704f |
0 10 * * 4 |
telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — cliprox/terra provider connection auth expired; shares the pin with 1b23c042e17d, 79f9bb41ecf5 |
| Daily Birthday Checker | a432898b55a4 |
30 7 * * * |
telegram:1793951355 | — | birthday_check.py |
|
| Daily Marketing Job Radar | dca8482e4f76 |
0 7 * * 1-5 |
telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 MODEL_NOT_IN_PLAN ×6 — see Known Issues |
| DHL Tracking - Parcel 8052838382 | 62619aa6bcb3 |
0 8,11,14,17,20 * * * |
telegram:1793951355 | — | dhl_tracking_monitor.py |
no-agent; added 2026-09-24 |
| Enhanced Weekend Adventure Planner | 1b23c042e17d |
0 8 * * 5 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30 (was bare telegram → home DM); delivery_failed flag is stale. Runs on cliprox/terra — same pin that failed 2026-10-01 |
| Evening Stand-Down Briefing | e530c89f2be0 |
45 18 * * 1-5 |
telegram:-1004321904721 | — | evening_standdown.py |
no-agent |
| Facebook Birthday Monthly Scrape | 0e753e9720fd |
0 9 1 * * |
telegram | — | — | ⚠ HTTP 400 ×1 (2026-10-01) — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare telegram → home DM; retarget to a group topic |
| Facebook Birthday Weekly Registry Sync | fad5a86979d4 |
0 3 * * 1 |
telegram | — | birthday_check.py |
⚠ HTTP 400 ×3 — nvidia/deepseek-ai/deepseek-v4-flash not in the active NVIDIA catalog; bare telegram target should move to a group topic |
| FreshRSS Midday News Briefing | ecc1fc13655a |
0 12 * * * |
telegram:-1004321904721:346 | — | freshrss_intelligence_v2.py |
|
| GitHub Trending Radar — Weekly Roundup | c0a9fa314542 |
0 9 * * 1 |
telegram:1793951355 | — | — | ⚠ Telegram bot token in github-trending-radar/config.json revoked (getMe → 401); digest builds, push fails |
| Gmail Action Digest | 08eb86f02d37 |
30 8 * * * |
telegram:-1004321904721:notifications | — | gmail_action_digest_v3.py |
|
| Important-mail monitor | d40085da631c |
every 30m |
telegram:1793951355 | — | _cron_silent_monitor.py |
disabled since 2026-08-29 (Gmail read-state incident) |
| Job Tracker Dashboard Refresh | 2fd50c87b0d0 |
0 */2 * * * |
telegram:-1004321904721:1030 | — | dashboard_watchdog.py |
no-agent |
| Mnemosyne Auto-Consolidate | 0c771463c3cd |
every 30m |
local | — | consolidate_mnemosyne.sh |
no-agent |
| Mnemosyne DB Backup | e5ed215fc91c |
0 3 * * 0 |
telegram:1793951355 | — | mnemosyne-backup.sh |
no-agent |
| Mnemosyne Embedding Coverage Monitor | b63ed5817212 |
0 */6 * * * |
telegram:-1004321904721:102 | — | check_embedding_coverage.py |
Alerts if coverage < 95% |
| Mnemosyne Health Monitor | ea79a2107e36 |
0 */12 * * * |
telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Watchdog | 96eb95ce12f2 |
*/15 * * * * |
telegram:-1004321904721:102 | — | mnemosyne-watchdog.py |
no-agent, silent ≤10 conflicts; also runs hermes-runtime-deps-guard.sh every 15 min (added 2026-09-30) |
| Mosquito Tracker | 1b31187397a5 |
0 7,15,22 * * * |
telegram:-1004321904721:273 | — | mosquito_tracker.py |
Excludes 23:00–07:00 |
| Perth New Restaurant & Cafe Openings (Weekly) | c8d52904e1d6 |
0 18 * * 4 |
telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified delivered, status ok |
| Political News Digest (Mon/Thu) | 0e519874d280 |
0 8 * * 1,4 |
telegram:-1004321904721:346 | — | — | |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
every 5m |
origin | — | agentmail-rhino-watch.sh |
disabled since 2026-07-27 (legacy) |
| Tech & AI Newsletter Digest (Analytical) | 85edf6772bca |
0 7 * * * |
telegram:-1004321904721:346 | — | — | |
| Uber Eats Monthly Spend Report | 066bb5191280 |
0 2 1 * * |
telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (configured thread_id 34795 for telegram:1793951355 was not found) — the 2026-09 report generated fine, only the send failed. Bare telegram → home DM; retarget to a group topic |
| Weekly Review | 79f9bb41ecf5 |
0 18 * * 0 |
origin | weekly-review | — | runs on cliprox/terra — same pin that failed 2026-10-01 |
| weekly-save-restricted-bot-restart | a688ed7b7ab0 |
0 3 * * 0 |
local | — | restart-save-restricted-bot.sh |
no-agent; script restored 2026-09-30 |
| Wiki Audit — Nightly | cbfe434a3b5e |
59 23 * * * |
local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
0 3 * * * |
local | — | wiki-maintain.sh |
⚠ HTTP 410 ×8 — qwen3.5:397b retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
| 🦝 Hermes Config Backup to Gitea (Daily) | e27da6c34bbb |
0 11 * * * |
origin | — | hermes-config-backup.sh |
no-agent |
Removed Jobs
| Name | Removed | Reason |
|---|---|---|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job d5516d0a5147 removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | d40085da631c |
2026-08-29 | Paused during Gmail read-state incident. See incidents/2026-08-29-gmail-read-state. |
| Rhino AgentMail Inbox Watcher | f77b7e46bd09 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | 0a7f933d06e2 |
2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
Known Issues
Resolved 2026-09-30
- Telegram thread delivery failures — root cause found and fixed: jobs
fad5a86979d4,1b23c042e17d,c8d52904e1d6reportedconfigured thread_id 34795 for telegram:1793951355 was not found/Message thread not found. The target was wrong, not the topic: a baredeliver: "telegram"resolves to the home DM (1793951355), where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong).1b23c042e17dandc8d52904e1d6were retargeted totelegram:-1004321904721:102and verified end-to-end. See systems/messaging-integrations. - Managed-runtime dependency failure (2026-09-28 → 30): Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause wasModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2, run every 15 min by job96eb95ce12f2. All managed venvs verified. See incidents/2026-09-28-hermes-managed-runtime-deps. weekly-save-restricted-bot-restartmissing script (2026-08-30): Previously failed with "Script not found:restart-save-restricted-bot.sh". The script existed at/home/hermes/scripts/but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job nowok, streak 0.- Residual pre-fix dispatch errors cleared:
cbfe434a3b5ereturned tookon its next run as predicted. The1a8e90ea2271error of the same era also cleared — its current failure is a different, upstream problem (below).
Open
- Daily Marketing Job Radar — 403
MODEL_NOT_IN_PLAN(2026-09-26 → 10-01, streak 6): Jobdca8482e4f76is pinned tocustom:omni-101/gpt-5.6-terra; the upstream (command-code) rejects it withGPT-5.6 Terra available in Pro and above plans. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See runbooks/job-radar. cliprox/terraprovider connection auth expired — NEW 2026-10-01 (streak 1): Job06a0a1c8704f(AI Newsletter Curation for Patrick) failed withHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard(providercustom:Omniroute:20128, modelcliprox/terra). Its failure notice reached topic 346; no digest was produced. This is a credential/connection problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. Two more enabled jobs run the same pin (1b23c042e17dEnhanced Weekend Adventure Planner,79f9bb41ecf5Weekly Review) and are expected to fail on their next runs. Re-authenticate the OmniRoute connection before repinning anything.- wiki-stocktake-maintenance — HTTP 410, model retired (streak 8): Job
1a8e90ea2271is pinned toollama-cloud / qwen3.5:397b, which Ollama retired on 2026-09-25 (qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT). It cannot self-recover — it needs a new model pin. Until then the daily 03:00 stocktake (skills-index regeneration, freshness audit, Quartz rebuild, Gitea commit) does not run. The page most visibly affected is systems/skills-index, whose last generated run was 2026-09-27 (341 skills) against 344 skills live on 2026-10-02. - Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1): Job
fad5a86979d4(weekly) is pinned tonvidia/deepseek-ai/deepseek-v4-flash; the provider reports the model isnot available in the active live catalog for provider 'nvidia'. Confirmed as a shared-pin issue on 2026-10-01, when sibling job0e753e9720fd(monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (smart-route) instead of a raw model name. - GitHub Trending Radar — revoked Telegram bot token: Job
c0a9fa314542builds its digest successfully, then fails the Telegram push (getMe→ 401 Unauthorized). The token lives in/home/hermes/workspace/github-trending-radar/config.jsonand needs replacing from @BotFather. Digest content is recoverable from the script's--dry-runoutput in the interim. - Stale provider key
custom:omni(2 jobs): the provider was removed fromcustom_providers(the live 2026-10-01 set islitellm,guacano,Omniroute:20128,command-code,omni-101). Jobse530c89f2be0andd40085da631cstill name it; both runsmart-route, and the defaultcustomprovider resolves to the same base URL (http://100.78.63.20:20128), so no failure has been observed — but the references should be normalised tocustom:omni-101. - Bare
telegramdelivery targets (3 jobs) — now demonstrably breaking sends:0e753e9720fd,fad5a86979d4,066bb5191280deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical:066bb5191280(02:11) and0e753e9720fd(09:01) both recordeddelivery_outcome: failedwithconfigured thread_id 34795 for telegram:1793951355 was not found. Work generated correctly; the delivery did not. Retarget all three totelegram:-1004321904721:<topic>. See systems/messaging-integrations. - Two wiki trees, drifted in both directions (NEW 2026-10-02):
/home/hermes/workspace/agent-estate-wikiis the canonical source (Gitea remoteAnthony/agent-estate-wiki, Quartz build,wiki-maintain.shtarget)./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads — pages nested underwiki/, and no git remote at all. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged backwards (systems/ignis.md,systems/nextcloud.md, threeincidents/pages exist only in the vault;raw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdonly in canonical) — a naive one-way sync deletes pages. See systems/hermes-agent.
Historical
- Mnemosyne Watchdog incident (2026-08-19): Job
96eb95ce12f2cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See systems/mnemosyne. - Session storage failures (2026-08-10): Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See current-state.
- Model repinning (2026-08-08): 13 jobs repinned from dead/stale provider combos to
smart-routeorgpt-5.6-terra. - Wiki Audit model drift (2026-08-07): Global inference config change caused
RuntimeError: Skipped to prevent unintended spendon an unpinned job.