Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh

This commit is contained in:
Tony0410
2026-09-30 19:38:49 +08:00
parent 389de2119b
commit 58312cb434
8 changed files with 245 additions and 75 deletions
+19 -9
View File
@@ -3,27 +3,37 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-09-04 updated: 2026-09-30
review_after: 2026-10-22 verified_on: 2026-09-30
review_after: 2026-10-30
confidence: high confidence: high
tags: [system] tags: [system]
sources: [] sources: [live-config, live-cron-list, live-hermes-version]
--- ---
# Current State # Current State
Updated: 2026-09-04 Updated: 2026-09-30
## Recent Changes (2026-09-30)
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
- **Open — Telegram thread delivery failures:** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Content runs; delivery does not.
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
## Recent Changes (2026-09-02) ## Recent Changes (2026-09-02)
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]]. - **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
## Attested facts ## Attested facts
- **Config version:** 33 — Verified, `~/.hermes/config.yaml`, 2026-07-25 - **Config version:** `_config_version: 46` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously recorded as 33 on 2026-07-25)
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22 - **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22 - **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
- **Main provider:** `custom:litellm` → `http://litellm:4000/v1` — Verified, `~/.hermes/config.yaml`, 2026-07-22 - **Main provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `custom:litellm` → `litellm:4000`, then `custom:omni` → `omni.kangaroo-eel.ts.net`)
- **Default model:** `freellmapi-auto` — Verified, `~/.hermes/config.yaml`, 2026-07-22 - **Default model:** `smart-route` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `freellmapi-auto`)
- **Fallback/aux provider:** `custom:omni` → `http://omni:20128/v1` — Verified, config, 2026-08-13 (migrated from `custom:omniroute`, CT600) - **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` — Verified, config, 2026-09-30
- **OmniRoute host:** `omni.kangaroo-eel.ts.net` → `100.93.204.4` — Verified, DNS resolution, 2026-08-13 - **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` — Verified, config, 2026-09-30
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22 - **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
- **Memory enabled:** true — Verified, config, 2026-07-22 - **Memory enabled:** true — Verified, config, 2026-07-22
- **Streaming enabled:** true — Verified, config, 2026-07-22 - **Streaming enabled:** true — Verified, config, 2026-07-22
@@ -0,0 +1,94 @@
---
title: Hermes Managed-Runtime Dependency Failure
type: incident
status: resolved
created: 2026-09-30
updated: 2026-09-30
verified_on: 2026-09-30
confidence: high
tags: [hermes, cron, runtime, python, worker, incident]
sources: [live-jobs-json, live-guard-script, live-cron-list]
---
# Hermes Managed-Runtime Dependency Failure
**Window:** ~2026-09-28 → 2026-09-30 19:13 AWST
**Status:** Resolved — dependency guard installed, watchdog green
**Impact:** Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
## Symptoms
Affected jobs reported the same dispatch error, which points at the wrong layer:
```
Restart-safe cron worker dispatch failed ... cron external worker exited
before ownership acknowledgement (exit 1)
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
sys.modules after import of package 'cron', but prior to execution of
'cron.scheduler'
```
The `RuntimeWarning` is noise. The real error is further down the worker stderr.
## Root cause
Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interpreter and venv:
```
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
```
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
- `ModuleNotFoundError: No module named 'dotenv'` (459 occurrences in worker stderr)
- `ModuleNotFoundError: No module named 'mnemosyne'`
## Diagnosis
Read worker stderr, not just the job's error field:
```bash
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
hermes cron list # which jobs, which streaks
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
```
Test the live interpreter directly — never trust `*.dist-info`.
## Fix
`/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh` (v2, installed 2026-09-30 19:13 AWST):
- discovers **every** managed venv under `~/.hermes/installs/*/environments/*/venv` (v1 checked only 2)
- verifies `import dotenv` and `import mnemosyne` against each venv
- repairs missing packages with `pip install --target <venv site-packages>`
Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
## Pitfalls found while fixing
- `uv pip install --python <venv>/bin/python` **walks up** to the `~/.hermes/hermes-agent` workspace and installs there — the target venv stays empty. Use `pip install --target`.
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
## Verification (2026-09-30)
| Check | Result |
|---|---|
| Managed venvs importing `dotenv` + `mnemosyne` | 10/10 |
| Mnemosyne Watchdog `96eb95ce12f2` last_status | `ok` (19:15 AWST), streak 0 |
| Jobs recovered | 12 of 13 |
| Residual | `cbfe434a3b5e` (Wiki Audit) and `1a8e90ea2271` (wiki-stocktake) still carry the pre-fix error — both last ran **before** the 19:13 fix. Expected to clear on their next runs. |
## Preventive rules
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
## Related
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
- [[systems/mnemosyne]]
- [[runbooks/restart-hermes]]
+3 -2
View File
@@ -1,6 +1,6 @@
# Anthony's Systems Wiki # Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure, > Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-09-20 > model routing and automation. Last updated: 2026-09-30
## Start here ## Start here
- [[current-state]] — Current systems status - [[current-state]] — Current systems status
@@ -28,10 +28,11 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — 27 active cron jobs - [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled)
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily) - [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily)
## Incidents ## Incidents
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured - [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration - [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
+10
View File
@@ -257,3 +257,13 @@
- **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`). - **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`).
- **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`. - **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`.
- No secrets written. - No secrets written.
## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
- **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`.
- **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`.
- **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30.
- **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
- **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps.
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
- No secrets written.
+22 -1
View File
@@ -26,7 +26,7 @@ Run a verified Western Australian marketing-job search, deduplicate against the
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) | | Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
| Delivery | Telegram group `-1004321904721`, topic `1030` | | Delivery | Telegram group `-1004321904721`, topic `1030` |
| Skill | `job-search-automation` | | Skill | `job-search-automation` |
| Primary model | `openai-codex / gpt-5.6-terra` | | Primary model | `custom:omni-101` / `gpt-5.6-terra` (upstream `command-code`) — ⚠ 403 `MODEL_NOT_IN_PLAN` since 2026-09-26 |
| First fallback | `opencode-go / mimo-v2.5` | | First fallback | `opencode-go / mimo-v2.5` |
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` | | Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
| Dashboard backend | `127.0.0.1:9099` | | Dashboard backend | `127.0.0.1:9099` |
@@ -241,6 +241,27 @@ tailscale serve status
- `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727` - `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727`
- `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727` - `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727`
## Model failure mode — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30)
Every run of the radar failed with:
```
[command-code/gpt-5.6-terra] [403]: MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans
```
**Cause:** the job is pinned to `gpt-5.6-terra` served through the `command-code` upstream. That upstream's plan no longer includes Terra, so the request is rejected before inference runs. Failure streak reached 5 by 2026-09-30 (last attempt 17:59 AWST).
**Diagnosis:**
```bash
hermes cron list # job dca8482e4f76 — check last_status + failure_streak
hermes cron run dca8482e4f76 # reproduce and capture the 403
```
**Recovery:** repin the job to a model the upstream still serves (e.g. an `opencode-go` or `smart-route` model), or keep Terra but route it through a provider whose plan includes it. Then re-run once and confirm `execution_success: true`.
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
+19 -17
View File
@@ -3,11 +3,11 @@ title: Hermes Agent
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-25 updated: 2026-09-30
verified_on: 2026-07-25 verified_on: 2026-09-30
confidence: high confidence: high
tags: [system, hermes, agent, llm] tags: [system, hermes, agent, llm]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [live-hermes-version, live-config]
--- ---
# Hermes Agent # Hermes Agent
@@ -16,15 +16,15 @@ sources: [raw/configs/hermes-config-sanitized.txt]
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status ## Current Status
- **Version:** v0.19.0 (2026.7.20), upstream `9ecacd6b` - **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30
- **Install method:** git - **Install method:** git
- **Python:** 3.11.15 - **Python:** 3.11.15
- **OpenAI SDK:** 2.24.0 - **Upstream status:** Update available (`hermes update`)
- **Upstream status:** Up to date
- **Install directory:** `/home/hermes/.hermes/hermes-agent` - **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions - **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
- **Binary:** `/home/hermes/.local/bin/hermes` - **Binary:** `/home/hermes/.local/bin/hermes`
- **Config:** `~/.hermes/config.yaml` (version 33) - **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (`~/.hermes/profiles/default/`) - **Profile:** `default` (`~/.hermes/profiles/default/`)
## Location ## Location
@@ -33,9 +33,9 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
- **Working directory:** `/home/hermes/workspace` - **Working directory:** `/home/hermes/workspace`
## Providers & Routing ## Providers & Routing
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1` - **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — verified 2026-09-30
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1` - **Default model:** `smart-route` — verified 2026-09-30
- **Default model:** `freellmapi-auto` - **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
- **Provider model count:** ~3430 (omniroute), ~1512 (litellm) - **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]]) - **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
- See [[model-providers]] for full breakdown. - See [[model-providers]] for full breakdown.
@@ -72,9 +72,9 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
- See [[terminal-backend]] for details. - See [[terminal-backend]] for details.
## Skills ## Skills
- **Total installed:** 66 - **Total installed:** 341 (per stocktake 2026-09-25)
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc. - Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
- See [[skills-index]] for categorized list. - See [[skills-index]] for categorized list (auto-regenerated daily).
## Multi-Profile Multiplexer ## Multi-Profile Multiplexer
@@ -105,7 +105,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- **Discord:** enabled, streaming true - **Discord:** enabled, streaming true
- **Slack:** enabled - **Slack:** enabled
- **Email:** connected - **Email:** connected
- **Mattermost:** connected - **Mattermost:** connected; gateway warning on 2026-08-15 was caused by a disabled system-level `hermes-gateway.service` unit, not a live duplicate gateway. Only the user-level service is active.
- **ntfy:** connected - **ntfy:** connected
- **Google Chat:** available (`hermes-google_chat`) - **Google Chat:** available (`hermes-google_chat`)
- **QQ:** available (`hermes-qqbot`) - **QQ:** available (`hermes-qqbot`)
@@ -113,9 +113,9 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- See [[messaging-integrations]] for details. - See [[messaging-integrations]] for details.
## Scheduled Tasks ## Scheduled Tasks
- **Cron jobs:** 25 active jobs - **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-09-30
- Delivery targets: Telegram (primary), Discord, origin, local - Delivery targets: Telegram (primary), Discord, origin, local
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher - Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
- See [[scheduled-tasks]] for full table. - See [[scheduled-tasks]] for full table.
## Dependencies ## Dependencies
@@ -126,9 +126,11 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
- Telegram bot token, Discord token, etc. (stored outside wiki) - Telegram bot token, Discord token, etc. (stored outside wiki)
## Known Issues ## Known Issues
- Router migration incomplete - Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- Daily Marketing Job Radar pinned to a model its upstream plan rejects (`gpt-5.6-terra` → 403 `MODEL_NOT_IN_PLAN`). See [[systems/scheduled-tasks]].
- Telegram thread delivery failures on `telegram:1793951355` (thread `34795` not found).
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
- Browser persistence not yet fully verified - Browser persistence not yet fully verified
- Obsidian Git plugin auto-sync caused conflicts during vault reorganization
## Related ## Related
- [[model-providers]] - [[model-providers]]
+12 -5
View File
@@ -14,10 +14,10 @@ sources: [raw/configs/hermes-config-sanitized.txt]
## Telegram ## Telegram
- **Status:** enabled, active - **Status:** enabled, active
- **allowed_chats:** `-1004321904721` (primary) - **allowed_chats:** `-1004321904721` (primary), `1793951355` (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
- **Rich messages:** true - **Rich messages:** true
- **Platform toolsets:** cli, telegram (full suite) - **Platform toolsets:** cli, telegram (full suite)
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721:1030`, `telegram:-1004321907D2:346`, `telegram:-1004321904721:273` - **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
- **Notes:** Primary delivery target for scheduled jobs - **Notes:** Primary delivery target for scheduled jobs
## Discord ## Discord
@@ -45,8 +45,15 @@ sources: [raw/configs/hermes-config-sanitized.txt]
- **Home Assistant:** homeassistant-platform enabled - **Home Assistant:** homeassistant-platform enabled
## Known Issues ## Known Issues
- Telegram thread_id `34795` for `telegram:1793951355` not found — one cron job delivered without thread_id - **Telegram thread delivery failures on `telegram:1793951355` (2026-09-30):** jobs `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reports `Message thread not found` (`last_status: delivery_failed`). Content generation succeeds — only the thread target is wrong. Fix: recreate the topic, or clear `thread_id` on those jobs. See [[systems/scheduled-tasks]].
## Multi-bot group chat (open question, raised 2026-09-30)
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
- **Preferred shape:** a Telegram supergroup with **one forum topic per bot**, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
- **Estate reachability:** CT460's Hermes serves the WebUI/API on `:9119` (locally fronted by nginx on `:19119`). Whether the other estate hosts expose `:9119` still needs checking before any fan-out design.
- **Status:** design question only — nothing has been deployed or changed.
## Related ## Related
- [[messaging-integrations]] - [[systems/hermes-agent]]
- [[scheduled-tasks]] - [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
+66 -41
View File
@@ -3,64 +3,89 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-09-19 updated: 2026-09-30
verified_on: 2026-09-19 verified_on: 2026-09-30
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list] sources: [live-cron-list, live-jobs-json]
--- ---
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Active jobs:** 26 (verified from live `hermes cron list`, 2026-09-19) - **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-09-30)
- **Previously 29** — 3 jobs removed or consolidated since 2026-08-16. - **Failing:** 4 (`dca8482e4f76` model-plan 403 ×5; `fad5a86979d4` thread; `cbfe434a3b5e` + `1a8e90ea2271` residual pre-fix dispatch error)
- **Delivery-only failures:** 2 (`c8d52904e1d6`, `1b23c042e17d` — content ran, Telegram thread missing)
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30)
## Job Table ## Job Table
| Name | Schedule | Delivery | Skills | Script | Notes | | Name | ID | Schedule | Delivery | Skills | Script | Notes |
|---|---|---|---|---|---| |---|---|---|---|---|---|---|
| Facebook Birthday Monthly Scrape | `0 9 1 * *` | telegram | — | — | ⚠ thread_id not found | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
| Facebook Birthday Weekly Registry Sync | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ thread_id not found | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
| Daily Birthday Checker | `30 7 * * *` | telegram | — | `birthday_check.py` | no-agent | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
| Daily Marketing Job Radar | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | Primary `gpt-5.6-terra`; see [[runbooks/job-radar]] | | Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×5 — see Known Issues |
| Perth New Restaurant & Cafe Openings (Weekly) | `0 18 * * 4` | telegram | — | — | | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
| AI Newsletter Curation for Patrick (Weekly) | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram | — | — | ⚠ `thread_id 34795` not found → delivered without thread |
| Enhanced Weekend Adventure Planner | `0 8 * * 5` | telegram | — | — | | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
| Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | telegram:-1004321904721:346 | — | — | | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | |
| Political News Digest (Mon/Thu) | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ `thread_id 34795` not found → delivered without thread |
| Uber Eats Monthly Spend Report | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
| Gmail Action Digest | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | | | GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | |
| FreshRSS Midday News Briefing | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
| Workday start reminder | `0 9 * * 1-5` | telegram | — | — | | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
| Important-mail monitor | `every 30m` | telegram:1793951355 | — | — | Silent when no important mail | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
| Mnemosyne Auto-Consolidate | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
| Mnemosyne DB Backup | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
| Mnemosyne Health Monitor | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
| Mnemosyne Watchdog | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts | | Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
| Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
| CT 221 Miner Watchdog | `every 15m` | telegram:1793951355 | pve-lxc-intrusion-detection | `ct221-miner-watchdog.sh` | Added 2026-08-07 after crypto miner incident | | Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
| Mosquito Tracker | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram | — | — | ⚠ delivery error "Message thread not found" |
| Weekly Review | `0 18 * * 0` | origin | weekly-review | — | | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
| Evening Stand-Down Briefing | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
| Job Tracker Dashboard Refresh | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
| Wiki Audit — Nightly | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
| GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | telegram:1793951355 | — | — | | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | |
| 🦝 Hermes Config Backup to Gitea (Daily) | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
| Review Agent Estate Phase One | once at 2026-08-08 18:33 | origin | — | — | ✅ Completed 2026-08-08. See [[current-state]]. | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
| wiki-stocktake-maintenance | `0 3 * * *` | local | — | `wiki-maintain.sh` | no-agent | | wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
## Removed Jobs ## Removed Jobs
| Name | Removed | Reason | | Name | Removed | Reason |
|---|---|---| |---|---|---|
| GitHub Trending Radar — Daily Digest | 2026-08-02 | Stopped by user. Weekly roundup still active. | | Workday start reminder | 2026-08-26 | No longer present in live cron list |
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
## Disabled Jobs (not removed)
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
## Known Issues ## Known Issues
- **Session storage failures (2026-08-10):** Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError ("session storage could not be written"). Disk at 83% (11GB free), `state.db` at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See [[current-state]] for details.
- **Wiki Audit cron model drift (2026-08-07):** Global inference config changed (`smart-route` → `deepseek-paid-smart`) and the job is unpinned, causing `RuntimeError: Skipped to prevent unintended spend`. Needs pinning: `cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>`. ### Resolved 2026-09-30
- **Facebook Birthday thread_id warnings:** Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread. - **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. 10/10 managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Model repinning (2026-08-08):** 13 cron jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. Dead combos included `freeapi/minimax-m2.7`, `freeapi/auto`, `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`. Jobs now running on valid model pins. - **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". Script restored 2026-09-30 19:17; job now `ok`, streak 0.
### Open
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30, streak 5):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change. See [[runbooks/job-radar]].
- **Telegram thread delivery failures (`telegram:1793951355`):** `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`; `c8d52904e1d6` reports `Message thread not found` (last_status `delivery_failed`). Content runs successfully — only the thread target is wrong. Fix: either recreate the topic or clear `thread_id` from the jobs.
- **Residual dispatch errors:** `cbfe434a3b5e` (streak 5) and `1a8e90ea2271` (streak 6) still show the pre-2026-09-30 worker error from runs made before the dependency guard was installed (23:59 AWST and 03:00 AWST respectively). Expected to clear on their next runs — re-check 2026-10-01 03:00.
### Historical
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
## Related ## Related
- [[systems/mnemosyne]] - [[systems/mnemosyne]]
- [[runbooks/job-radar]] - [[runbooks/job-radar]]
- [[systems/hermes-agent]]
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]