Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh
This commit is contained in:
+19
-9
@@ -3,27 +3,37 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-09-04
|
updated: 2026-09-30
|
||||||
review_after: 2026-10-22
|
verified_on: 2026-09-30
|
||||||
|
review_after: 2026-10-30
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
sources: []
|
sources: [live-config, live-cron-list, live-hermes-version]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-09-04
|
Updated: 2026-09-30
|
||||||
|
|
||||||
|
## Recent Changes (2026-09-30)
|
||||||
|
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
|
||||||
|
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
|
||||||
|
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
|
||||||
|
- **Open — Telegram thread delivery failures:** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Content runs; delivery does not.
|
||||||
|
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
|
||||||
|
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
|
||||||
|
|
||||||
## Recent Changes (2026-09-02)
|
## Recent Changes (2026-09-02)
|
||||||
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
|
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
|
||||||
|
|
||||||
## Attested facts
|
## Attested facts
|
||||||
- **Config version:** 33 — Verified, `~/.hermes/config.yaml`, 2026-07-25
|
- **Config version:** `_config_version: 46` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously recorded as 33 on 2026-07-25)
|
||||||
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
|
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
|
||||||
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
|
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
|
||||||
- **Main provider:** `custom:litellm` → `http://litellm:4000/v1` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
- **Main provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `custom:litellm` → `litellm:4000`, then `custom:omni` → `omni.kangaroo-eel.ts.net`)
|
||||||
- **Default model:** `freellmapi-auto` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
- **Default model:** `smart-route` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `freellmapi-auto`)
|
||||||
- **Fallback/aux provider:** `custom:omni` → `http://omni:20128/v1` — Verified, config, 2026-08-13 (migrated from `custom:omniroute`, CT600)
|
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` — Verified, config, 2026-09-30
|
||||||
- **OmniRoute host:** `omni.kangaroo-eel.ts.net` → `100.93.204.4` — Verified, DNS resolution, 2026-08-13
|
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` — Verified, config, 2026-09-30
|
||||||
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
|
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
|
||||||
- **Memory enabled:** true — Verified, config, 2026-07-22
|
- **Memory enabled:** true — Verified, config, 2026-07-22
|
||||||
- **Streaming enabled:** true — Verified, config, 2026-07-22
|
- **Streaming enabled:** true — Verified, config, 2026-07-22
|
||||||
|
|||||||
@@ -0,0 +1,94 @@
|
|||||||
|
---
|
||||||
|
title: Hermes Managed-Runtime Dependency Failure
|
||||||
|
type: incident
|
||||||
|
status: resolved
|
||||||
|
created: 2026-09-30
|
||||||
|
updated: 2026-09-30
|
||||||
|
verified_on: 2026-09-30
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, cron, runtime, python, worker, incident]
|
||||||
|
sources: [live-jobs-json, live-guard-script, live-cron-list]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes Managed-Runtime Dependency Failure
|
||||||
|
|
||||||
|
**Window:** ~2026-09-28 → 2026-09-30 19:13 AWST
|
||||||
|
**Status:** Resolved — dependency guard installed, watchdog green
|
||||||
|
**Impact:** Cron jobs dispatched through the external worker; Mnemosyne Watchdog failed 176 consecutive runs
|
||||||
|
|
||||||
|
## Symptoms
|
||||||
|
|
||||||
|
Affected jobs reported the same dispatch error, which points at the wrong layer:
|
||||||
|
|
||||||
|
```
|
||||||
|
Restart-safe cron worker dispatch failed ... cron external worker exited
|
||||||
|
before ownership acknowledgement (exit 1)
|
||||||
|
worker stderr: <frozen runpy>:130: RuntimeWarning: 'cron.scheduler' found in
|
||||||
|
sys.modules after import of package 'cron', but prior to execution of
|
||||||
|
'cron.scheduler'
|
||||||
|
```
|
||||||
|
|
||||||
|
The `RuntimeWarning` is noise. The real error is further down the worker stderr.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
Hermes migrated to **uv-managed runtimes**. Each runtime gets its own interpreter and venv:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv # 10 venvs at time of incident
|
||||||
|
```
|
||||||
|
|
||||||
|
The migration did not carry every dependency into every managed venv. Workers launched against a venv missing `dotenv` and `mnemosyne` died during import — *before* acknowledging ownership of the job. The failure therefore surfaced as a scheduler/dispatch bug while the real cause was:
|
||||||
|
|
||||||
|
- `ModuleNotFoundError: No module named 'dotenv'` (459 occurrences in worker stderr)
|
||||||
|
- `ModuleNotFoundError: No module named 'mnemosyne'`
|
||||||
|
|
||||||
|
## Diagnosis
|
||||||
|
|
||||||
|
Read worker stderr, not just the job's error field:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -rl "ModuleNotFoundError" ~/.hermes/logs/ # find the worker log
|
||||||
|
hermes cron list # which jobs, which streaks
|
||||||
|
ls -d ~/.hermes/installs/*/environments/*/venv # how many runtimes exist
|
||||||
|
```
|
||||||
|
|
||||||
|
Test the live interpreter directly — never trust `*.dist-info`.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
`/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh` (v2, installed 2026-09-30 19:13 AWST):
|
||||||
|
|
||||||
|
- discovers **every** managed venv under `~/.hermes/installs/*/environments/*/venv` (v1 checked only 2)
|
||||||
|
- verifies `import dotenv` and `import mnemosyne` against each venv
|
||||||
|
- repairs missing packages with `pip install --target <venv site-packages>`
|
||||||
|
|
||||||
|
Wiring: the **Mnemosyne Watchdog** job (`96eb95ce12f2`, `*/15 * * * *`) runs the guard every 15 minutes, so a future managed-runtime migration self-heals within 15 minutes.
|
||||||
|
|
||||||
|
## Pitfalls found while fixing
|
||||||
|
|
||||||
|
- `uv pip install --python <venv>/bin/python` **walks up** to the `~/.hermes/hermes-agent` workspace and installs there — the target venv stays empty. Use `pip install --target`.
|
||||||
|
- Stale `*.dist-info` directories make `uv` skip a package as "already installed" even when the module is not importable.
|
||||||
|
- `write_file` silently no-ops on `.hermes/scripts/` paths (Hermes library bug). Stage the script in `/tmp` and `install` it into place.
|
||||||
|
|
||||||
|
## Verification (2026-09-30)
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Managed venvs importing `dotenv` + `mnemosyne` | 10/10 |
|
||||||
|
| Mnemosyne Watchdog `96eb95ce12f2` last_status | `ok` (19:15 AWST), streak 0 |
|
||||||
|
| Jobs recovered | 12 of 13 |
|
||||||
|
| Residual | `cbfe434a3b5e` (Wiki Audit) and `1a8e90ea2271` (wiki-stocktake) still carry the pre-fix error — both last ran **before** the 19:13 fix. Expected to clear on their next runs. |
|
||||||
|
|
||||||
|
## Preventive rules
|
||||||
|
|
||||||
|
- Treat "worker exited before ownership acknowledgement" as a **dependency/import** failure first. Read worker stderr before touching provider or model pins.
|
||||||
|
- After any Hermes runtime or Python upgrade, re-run the guard and confirm 10/10 managed venvs import `dotenv` and `mnemosyne`.
|
||||||
|
- Mass-cron-failure triage order: live interpreter imports → worker stderr → runtime layout → job logic. See [[runbooks/restart-hermes]] and [[systems/scheduled-tasks]].
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[systems/scheduled-tasks]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[runbooks/restart-hermes]]
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-09-20
|
> model routing and automation. Last updated: 2026-09-30
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
- [[current-state]] — Current systems status
|
- [[current-state]] — Current systems status
|
||||||
@@ -28,10 +28,11 @@
|
|||||||
|
|
||||||
### Integrations
|
### Integrations
|
||||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
- [[systems/scheduled-tasks]] — 27 active cron jobs
|
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled)
|
||||||
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily)
|
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily)
|
||||||
|
|
||||||
## Incidents
|
## Incidents
|
||||||
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
||||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
||||||
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
||||||
|
|
||||||
|
|||||||
@@ -257,3 +257,13 @@
|
|||||||
- **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`).
|
- **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`).
|
||||||
- **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`.
|
- **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`.
|
||||||
- No secrets written.
|
- No secrets written.
|
||||||
|
|
||||||
|
## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
|
||||||
|
- **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`.
|
||||||
|
- **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`.
|
||||||
|
- **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30.
|
||||||
|
- **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
|
||||||
|
- **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps.
|
||||||
|
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
|
||||||
|
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
|
||||||
|
- No secrets written.
|
||||||
|
|||||||
+22
-1
@@ -26,7 +26,7 @@ Run a verified Western Australian marketing-job search, deduplicate against the
|
|||||||
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
|
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
|
||||||
| Delivery | Telegram group `-1004321904721`, topic `1030` |
|
| Delivery | Telegram group `-1004321904721`, topic `1030` |
|
||||||
| Skill | `job-search-automation` |
|
| Skill | `job-search-automation` |
|
||||||
| Primary model | `openai-codex / gpt-5.6-terra` |
|
| Primary model | `custom:omni-101` / `gpt-5.6-terra` (upstream `command-code`) — ⚠ 403 `MODEL_NOT_IN_PLAN` since 2026-09-26 |
|
||||||
| First fallback | `opencode-go / mimo-v2.5` |
|
| First fallback | `opencode-go / mimo-v2.5` |
|
||||||
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
|
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
|
||||||
| Dashboard backend | `127.0.0.1:9099` |
|
| Dashboard backend | `127.0.0.1:9099` |
|
||||||
@@ -241,6 +241,27 @@ tailscale serve status
|
|||||||
- `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727`
|
- `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727`
|
||||||
- `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727`
|
- `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727`
|
||||||
|
|
||||||
|
## Model failure mode — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30)
|
||||||
|
|
||||||
|
Every run of the radar failed with:
|
||||||
|
|
||||||
|
```
|
||||||
|
[command-code/gpt-5.6-terra] [403]: MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans
|
||||||
|
```
|
||||||
|
|
||||||
|
**Cause:** the job is pinned to `gpt-5.6-terra` served through the `command-code` upstream. That upstream's plan no longer includes Terra, so the request is rejected before inference runs. Failure streak reached 5 by 2026-09-30 (last attempt 17:59 AWST).
|
||||||
|
|
||||||
|
**Diagnosis:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
hermes cron list # job dca8482e4f76 — check last_status + failure_streak
|
||||||
|
hermes cron run dca8482e4f76 # reproduce and capture the 403
|
||||||
|
```
|
||||||
|
|
||||||
|
**Recovery:** repin the job to a model the upstream still serves (e.g. an `opencode-go` or `smart-route` model), or keep Terra but route it through a provider whose plan includes it. Then re-run once and confirm `execution_success: true`.
|
||||||
|
|
||||||
|
Do **not** allow a silent fallback — the radar must surface a model failure rather than emit an empty "no jobs found" digest.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[systems/scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
|
|||||||
+19
-17
@@ -3,11 +3,11 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-25
|
updated: 2026-09-30
|
||||||
verified_on: 2026-07-25
|
verified_on: 2026-09-30
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [live-hermes-version, live-config]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Hermes Agent
|
# Hermes Agent
|
||||||
@@ -16,15 +16,15 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
||||||
|
|
||||||
## Current Status
|
## Current Status
|
||||||
- **Version:** v0.19.0 (2026.7.20), upstream `9ecacd6b`
|
- **Version:** v0.21.5+3837.gbfda74c (2026.9.24), upstream `27062c34` — verified 2026-09-30
|
||||||
- **Install method:** git
|
- **Install method:** git
|
||||||
- **Python:** 3.11.15
|
- **Python:** 3.11.15
|
||||||
- **OpenAI SDK:** 2.24.0
|
- **Upstream status:** Update available (`hermes update`)
|
||||||
- **Upstream status:** Up to date
|
|
||||||
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||||
|
- **Managed runtimes:** 10 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv`. Dependency guard `hermes-runtime-deps-guard.sh` verifies `dotenv` + `mnemosyne` in every venv every 15 min — see [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
||||||
- **Binary:** `/home/hermes/.local/bin/hermes`
|
- **Binary:** `/home/hermes/.local/bin/hermes`
|
||||||
- **Config:** `~/.hermes/config.yaml` (version 33)
|
- **Config:** `~/.hermes/config.yaml`
|
||||||
- **Profile:** `default` (`~/.hermes/profiles/default/`)
|
- **Profile:** `default` (`~/.hermes/profiles/default/`)
|
||||||
|
|
||||||
## Location
|
## Location
|
||||||
@@ -33,9 +33,9 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
- **Working directory:** `/home/hermes/workspace`
|
- **Working directory:** `/home/hermes/workspace`
|
||||||
|
|
||||||
## Providers & Routing
|
## Providers & Routing
|
||||||
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
|
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — verified 2026-09-30
|
||||||
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1`
|
- **Default model:** `smart-route` — verified 2026-09-30
|
||||||
- **Default model:** `freellmapi-auto`
|
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
|
||||||
- **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
|
- **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
|
||||||
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
||||||
- See [[model-providers]] for full breakdown.
|
- See [[model-providers]] for full breakdown.
|
||||||
@@ -72,9 +72,9 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
- See [[terminal-backend]] for details.
|
- See [[terminal-backend]] for details.
|
||||||
|
|
||||||
## Skills
|
## Skills
|
||||||
- **Total installed:** 66
|
- **Total installed:** 341 (per stocktake 2026-09-25)
|
||||||
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
||||||
- See [[skills-index]] for categorized list.
|
- See [[skills-index]] for categorized list (auto-regenerated daily).
|
||||||
|
|
||||||
## Multi-Profile Multiplexer
|
## Multi-Profile Multiplexer
|
||||||
|
|
||||||
@@ -105,7 +105,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- **Discord:** enabled, streaming true
|
- **Discord:** enabled, streaming true
|
||||||
- **Slack:** enabled
|
- **Slack:** enabled
|
||||||
- **Email:** connected
|
- **Email:** connected
|
||||||
- **Mattermost:** connected
|
- **Mattermost:** connected; gateway warning on 2026-08-15 was caused by a disabled system-level `hermes-gateway.service` unit, not a live duplicate gateway. Only the user-level service is active.
|
||||||
- **ntfy:** connected
|
- **ntfy:** connected
|
||||||
- **Google Chat:** available (`hermes-google_chat`)
|
- **Google Chat:** available (`hermes-google_chat`)
|
||||||
- **QQ:** available (`hermes-qqbot`)
|
- **QQ:** available (`hermes-qqbot`)
|
||||||
@@ -113,9 +113,9 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- See [[messaging-integrations]] for details.
|
- See [[messaging-integrations]] for details.
|
||||||
|
|
||||||
## Scheduled Tasks
|
## Scheduled Tasks
|
||||||
- **Cron jobs:** 25 active jobs
|
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-09-30
|
||||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||||
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher
|
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
|
||||||
- See [[scheduled-tasks]] for full table.
|
- See [[scheduled-tasks]] for full table.
|
||||||
|
|
||||||
## Dependencies
|
## Dependencies
|
||||||
@@ -126,9 +126,11 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- Telegram bot token, Discord token, etc. (stored outside wiki)
|
- Telegram bot token, Discord token, etc. (stored outside wiki)
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- Router migration incomplete
|
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- Daily Marketing Job Radar pinned to a model its upstream plan rejects (`gpt-5.6-terra` → 403 `MODEL_NOT_IN_PLAN`). See [[systems/scheduled-tasks]].
|
||||||
|
- Telegram thread delivery failures on `telegram:1793951355` (thread `34795` not found).
|
||||||
|
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
|
||||||
- Browser persistence not yet fully verified
|
- Browser persistence not yet fully verified
|
||||||
- Obsidian Git plugin auto-sync caused conflicts during vault reorganization
|
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[model-providers]]
|
- [[model-providers]]
|
||||||
|
|||||||
@@ -14,10 +14,10 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
|
|
||||||
## Telegram
|
## Telegram
|
||||||
- **Status:** enabled, active
|
- **Status:** enabled, active
|
||||||
- **allowed_chats:** `-1004321904721` (primary)
|
- **allowed_chats:** `-1004321904721` (primary), `1793951355` (DM), plus the remaining chat IDs required by both multiplexed profiles on the default adapter — verified 2026-09-30
|
||||||
- **Rich messages:** true
|
- **Rich messages:** true
|
||||||
- **Platform toolsets:** cli, telegram (full suite)
|
- **Platform toolsets:** cli, telegram (full suite)
|
||||||
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721:1030`, `telegram:-1004321907D2:346`, `telegram:-1004321904721:273`
|
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
|
||||||
- **Notes:** Primary delivery target for scheduled jobs
|
- **Notes:** Primary delivery target for scheduled jobs
|
||||||
|
|
||||||
## Discord
|
## Discord
|
||||||
@@ -45,8 +45,15 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
- **Home Assistant:** homeassistant-platform enabled
|
- **Home Assistant:** homeassistant-platform enabled
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- Telegram thread_id `34795` for `telegram:1793951355` not found — one cron job delivered without thread_id
|
- **Telegram thread delivery failures on `telegram:1793951355` (2026-09-30):** jobs `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reports `Message thread not found` (`last_status: delivery_failed`). Content generation succeeds — only the thread target is wrong. Fix: recreate the topic, or clear `thread_id` on those jobs. See [[systems/scheduled-tasks]].
|
||||||
|
|
||||||
|
## Multi-bot group chat (open question, raised 2026-09-30)
|
||||||
|
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
|
||||||
|
- **Preferred shape:** a Telegram supergroup with **one forum topic per bot**, so each agent posts into its own thread and no two bots share a message stream. Mattermost was floated and rejected.
|
||||||
|
- **Estate reachability:** CT460's Hermes serves the WebUI/API on `:9119` (locally fronted by nginx on `:19119`). Whether the other estate hosts expose `:9119` still needs checking before any fan-out design.
|
||||||
|
- **Status:** design question only — nothing has been deployed or changed.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[messaging-integrations]]
|
- [[systems/hermes-agent]]
|
||||||
- [[scheduled-tasks]]
|
- [[systems/scheduled-tasks]]
|
||||||
|
- [[infrastructure/tailscale]]
|
||||||
|
|||||||
+66
-41
@@ -3,64 +3,89 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-09-19
|
updated: 2026-09-30
|
||||||
verified_on: 2026-09-19
|
verified_on: 2026-09-30
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list]
|
sources: [live-cron-list, live-jobs-json]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
- **Active jobs:** 26 (verified from live `hermes cron list`, 2026-09-19)
|
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-09-30)
|
||||||
- **Previously 29** — 3 jobs removed or consolidated since 2026-08-16.
|
- **Failing:** 4 (`dca8482e4f76` model-plan 403 ×5; `fad5a86979d4` thread; `cbfe434a3b5e` + `1a8e90ea2271` residual pre-fix dispatch error)
|
||||||
|
- **Delivery-only failures:** 2 (`c8d52904e1d6`, `1b23c042e17d` — content ran, Telegram thread missing)
|
||||||
|
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30)
|
||||||
|
|
||||||
## Job Table
|
## Job Table
|
||||||
|
|
||||||
| Name | Schedule | Delivery | Skills | Script | Notes |
|
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|
|
||||||
| Facebook Birthday Monthly Scrape | `0 9 1 * *` | telegram | — | — | ⚠ thread_id not found |
|
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||||||
| Facebook Birthday Weekly Registry Sync | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ thread_id not found |
|
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Daily Birthday Checker | `30 7 * * *` | telegram | — | `birthday_check.py` | no-agent |
|
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||||
| Daily Marketing Job Radar | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | Primary `gpt-5.6-terra`; see [[runbooks/job-radar]] |
|
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×5 — see Known Issues |
|
||||||
| Perth New Restaurant & Cafe Openings (Weekly) | `0 18 * * 4` | telegram | — | — | |
|
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
||||||
| AI Newsletter Curation for Patrick (Weekly) | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
|
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram | — | — | ⚠ `thread_id 34795` not found → delivered without thread |
|
||||||
| Enhanced Weekend Adventure Planner | `0 8 * * 5` | telegram | — | — | |
|
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||||
| Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | |
|
||||||
| Political News Digest (Mon/Thu) | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ `thread_id 34795` not found → delivered without thread |
|
||||||
| Uber Eats Monthly Spend Report | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||||
| Gmail Action Digest | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
||||||
| FreshRSS Midday News Briefing | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||||
| Workday start reminder | `0 9 * * 1-5` | telegram | — | — | |
|
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||||
| Important-mail monitor | `every 30m` | telegram:1793951355 | — | — | Silent when no important mail |
|
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||||
| Mnemosyne Auto-Consolidate | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
| Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||||
| Mnemosyne DB Backup | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
| Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||||
| Mnemosyne Health Monitor | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
||||||
| Mnemosyne Watchdog | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts |
|
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||||
| Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
||||||
| CT 221 Miner Watchdog | `every 15m` | telegram:1793951355 | pve-lxc-intrusion-detection | `ct221-miner-watchdog.sh` | Added 2026-08-07 after crypto miner incident |
|
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||||
| Mosquito Tracker | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram | — | — | ⚠ delivery error "Message thread not found" |
|
||||||
| Weekly Review | `0 18 * * 0` | origin | weekly-review | — | |
|
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Evening Stand-Down Briefing | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||||
| Job Tracker Dashboard Refresh | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Wiki Audit — Nightly | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
||||||
| GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | |
|
||||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||||
| Review Agent Estate Phase One | once at 2026-08-08 18:33 | origin | — | — | ✅ Completed 2026-08-08. See [[current-state]]. |
|
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||||
| wiki-stocktake-maintenance | `0 3 * * *` | local | — | `wiki-maintain.sh` | no-agent |
|
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
|
||||||
|
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||||
|
|
||||||
## Removed Jobs
|
## Removed Jobs
|
||||||
| Name | Removed | Reason |
|
| Name | Removed | Reason |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| GitHub Trending Radar — Daily Digest | 2026-08-02 | Stopped by user. Weekly roundup still active. |
|
| Workday start reminder | 2026-08-26 | No longer present in live cron list |
|
||||||
|
| CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected |
|
||||||
|
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
|
||||||
|
|
||||||
|
## Disabled Jobs (not removed)
|
||||||
|
| Name | ID | Disabled since | Reason |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
|
||||||
|
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||||
|
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. |
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- **Session storage failures (2026-08-10):** Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError ("session storage could not be written"). Disk at 83% (11GB free), `state.db` at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See [[current-state]] for details.
|
|
||||||
- **Wiki Audit cron model drift (2026-08-07):** Global inference config changed (`smart-route` → `deepseek-paid-smart`) and the job is unpinned, causing `RuntimeError: Skipped to prevent unintended spend`. Needs pinning: `cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>`.
|
### Resolved 2026-09-30
|
||||||
- **Facebook Birthday thread_id warnings:** Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread.
|
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. 10/10 managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
- **Model repinning (2026-08-08):** 13 cron jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. Dead combos included `freeapi/minimax-m2.7`, `freeapi/auto`, `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`. Jobs now running on valid model pins.
|
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". Script restored 2026-09-30 19:17; job now `ok`, streak 0.
|
||||||
|
|
||||||
|
### Open
|
||||||
|
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30, streak 5):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change. See [[runbooks/job-radar]].
|
||||||
|
- **Telegram thread delivery failures (`telegram:1793951355`):** `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`; `c8d52904e1d6` reports `Message thread not found` (last_status `delivery_failed`). Content runs successfully — only the thread target is wrong. Fix: either recreate the topic or clear `thread_id` from the jobs.
|
||||||
|
- **Residual dispatch errors:** `cbfe434a3b5e` (streak 5) and `1a8e90ea2271` (streak 6) still show the pre-2026-09-30 worker error from runs made before the dependency guard was installed (23:59 AWST and 03:00 AWST respectively). Expected to clear on their next runs — re-check 2026-10-01 03:00.
|
||||||
|
|
||||||
|
### Historical
|
||||||
|
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||||
|
- **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].
|
||||||
|
- **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`.
|
||||||
|
- **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[runbooks/job-radar]]
|
- [[runbooks/job-radar]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||||
|
|||||||
Reference in New Issue
Block a user