wiki: nightly audit 2026-10-01 — cron fleet re-verified, Telegram delivery root cause, memory hygiene
This commit is contained in:
+13
-4
@@ -3,8 +3,8 @@ title: Current State
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-09-30
|
updated: 2026-10-01
|
||||||
verified_on: 2026-09-30
|
verified_on: 2026-10-01
|
||||||
review_after: 2026-10-30
|
review_after: 2026-10-30
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system]
|
tags: [system]
|
||||||
@@ -12,14 +12,23 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
|||||||
---
|
---
|
||||||
|
|
||||||
# Current State
|
# Current State
|
||||||
Updated: 2026-09-30
|
Updated: 2026-10-01
|
||||||
|
|
||||||
|
## Recent Changes (2026-10-01)
|
||||||
|
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **4 → 3**: `cbfe434a3b5e` returned to `ok` as predicted on 2026-09-30. The residual `1a8e90ea2271` dispatch error also cleared — its current failure is a different, upstream problem (below).
|
||||||
|
- **Telegram delivery failures root-caused and fixed:** the target, not the topic, was wrong — a bare `deliver: "telegram"` resolves to the **home DM (`1793951355`)**, where group topic IDs cannot exist, so sends hard-failed and the message was lost. `1b23c042e17d` and `c8d52904e1d6` retargeted to `telegram:-1004321904721:102` and verified. Three jobs still carry bare `telegram` targets (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). See [[systems/messaging-integrations]].
|
||||||
|
- **Three jobs pinned to models that no longer exist upstream — cannot self-recover:** `1a8e90ea2271` → `qwen3.5:397b` retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`, streak 5); `fad5a86979d4` → `nvidia/…/deepseek-v4-flash` absent from the live catalog (HTTP 400, streak 3). Repin to the `smart-route` alias. See [[systems/model-providers]].
|
||||||
|
- **GitHub Trending Radar — revoked Telegram bot token:** `c0a9fa314542` builds its digest then fails the push (`getMe` → 401). Token in `/home/hermes/workspace/github-trending-radar/config.json` needs replacing from @BotFather.
|
||||||
|
- **Memory hygiene pass (2026-09-30):** 36 gateway background-process notifications had been written into Mnemosyne as `[USER] [Anthony Martin]` memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts `hermes_cron_model_default` and the job-radar model corrected. See [[systems/mnemosyne]].
|
||||||
|
- **Provider stack rebuilt from live config:** 5 custom providers (`omni-101`, `Omniroute:20128`, `litellm`, `command-code`, `guacano`). The provider key `custom:omni` **no longer exists**; two cron jobs (`e530c89f2be0`, `d40085da631c`) still name it but have not failed, because the default `custom` block resolves to the same base URL.
|
||||||
|
- **Vision routing — request recorded, not applied:** Anthony asked on 2026-09-30 that vision calls use the local Ollama at `http://anthonys-mac-mini:11434` (gemma4). Live config still points at `nous / stepfun/step-3.7-flash:free` as at 2026-10-01.
|
||||||
|
|
||||||
## Recent Changes (2026-09-30)
|
## Recent Changes (2026-09-30)
|
||||||
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
|
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
|
||||||
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
|
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
|
||||||
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
|
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
|
||||||
- **Open — Telegram thread delivery failures:** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Content runs; delivery does not.
|
- **Open — Telegram thread delivery failures (root cause found 2026-10-01):** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Cause was the delivery target, not the topic — a bare `telegram` resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
|
||||||
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
|
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
|
||||||
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
|
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Anthony's Systems Wiki
|
# Anthony's Systems Wiki
|
||||||
> Operational knowledge for Anthony's agents, infrastructure,
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
> model routing and automation. Last updated: 2026-09-30
|
> model routing and automation. Last updated: 2026-10-01
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
- [[current-state]] — Current systems status
|
- [[current-state]] — Current systems status
|
||||||
|
|||||||
@@ -267,3 +267,14 @@
|
|||||||
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
|
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
|
||||||
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
|
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
|
||||||
- No secrets written.
|
- No secrets written.
|
||||||
|
|
||||||
|
## [2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
|
||||||
|
- **No new pages created** — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
|
||||||
|
- **Updated `systems/scheduled-tasks.md`** — table and counts re-verified against live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, ×5), `1a8e90ea2271` (HTTP 410, model retired upstream, ×7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: `1b23c042e17d` and `c8d52904e1d6` delivery targets now `telegram:-1004321904721:102`; `c0a9fa314542` flagged for a revoked Telegram bot token; bare-`telegram` targets noted on `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`. Known Issues restructured into Resolved / Open / Historical.
|
||||||
|
- **Updated `systems/messaging-integrations.md`** — new **Delivery target semantics** section: a bare `deliver: "telegram"` resolves to the home DM (`1793951355`), *not* to the primary group, so a group topic ID attached to it hard-fails with `Message thread not found` and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
|
||||||
|
- **Updated `systems/model-providers.md`** — provider stack rebuilt from live `config.yaml`: 5 custom providers (`omni-101` 5,384 models, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46); `custom:omni` recorded as removed. Default model corrected `resilient-free` → `smart-route`. New **Retired / Unavailable Models** table (retired `qwen3.5:397b`, plan-gated `gpt-5.6-terra`, catalog-missing `deepseek-v4-flash`) with the lesson that a raw model pin never self-recovers.
|
||||||
|
- **Updated `systems/mnemosyne.md`** — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as `[USER] [Anthony Martin]` memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
|
||||||
|
- **Updated `systems/hermes-agent.md`** — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
|
||||||
|
- **Updated `index.md` and `current-state.md`** — last-updated bumped to 2026-10-01 with a Recent Changes block.
|
||||||
|
- **Vault sync:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under `wiki/`, not at the tree root, and `index.md` sits at the vault root.
|
||||||
|
- No secrets written.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes Agent
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-09-30
|
updated: 2026-10-01
|
||||||
verified_on: 2026-09-30
|
verified_on: 2026-10-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, hermes, agent, llm]
|
tags: [system, hermes, agent, llm]
|
||||||
sources: [live-hermes-version, live-config]
|
sources: [live-hermes-version, live-config]
|
||||||
@@ -36,7 +36,9 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — verified 2026-09-30
|
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — verified 2026-09-30
|
||||||
- **Default model:** `smart-route` — verified 2026-09-30
|
- **Default model:** `smart-route` — verified 2026-09-30
|
||||||
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
|
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
|
||||||
- **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
|
- **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46
|
||||||
|
- **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01
|
||||||
|
- **Plugins enabled:** 11 — verified 2026-10-01
|
||||||
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
||||||
- See [[model-providers]] for full breakdown.
|
- See [[model-providers]] for full breakdown.
|
||||||
|
|
||||||
@@ -72,7 +74,7 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
|||||||
- See [[terminal-backend]] for details.
|
- See [[terminal-backend]] for details.
|
||||||
|
|
||||||
## Skills
|
## Skills
|
||||||
- **Total installed:** 341 (per stocktake 2026-09-25)
|
- **Total installed:** 344 `SKILL.md` files across 72 top-level skill directories (live count, 2026-10-01; the last full stocktake on 2026-09-25 recorded 341)
|
||||||
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
||||||
- See [[skills-index]] for categorized list (auto-regenerated daily).
|
- See [[skills-index]] for categorized list (auto-regenerated daily).
|
||||||
|
|
||||||
@@ -113,7 +115,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
- See [[messaging-integrations]] for details.
|
- See [[messaging-integrations]] for details.
|
||||||
|
|
||||||
## Scheduled Tasks
|
## Scheduled Tasks
|
||||||
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-09-30
|
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-10-01
|
||||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||||
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
|
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
|
||||||
- See [[scheduled-tasks]] for full table.
|
- See [[scheduled-tasks]] for full table.
|
||||||
@@ -127,8 +129,8 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
|||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
- Daily Marketing Job Radar pinned to a model its upstream plan rejects (`gpt-5.6-terra` → 403 `MODEL_NOT_IN_PLAN`). See [[systems/scheduled-tasks]].
|
- **Three cron jobs are pinned to models that no longer exist upstream and cannot self-recover** (`1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25, HTTP 410; `dca8482e4f76` → `gpt-5.6-terra` plan-gated, 403 `MODEL_NOT_IN_PLAN`; `fad5a86979d4` → `deepseek-v4-flash` absent from the NVIDIA live catalog, HTTP 400). Repinning to the `smart-route` alias is the durable fix. See [[systems/scheduled-tasks]] and [[systems/model-providers]].
|
||||||
- Telegram thread delivery failures on `telegram:1793951355` (thread `34795` not found).
|
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
|
||||||
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
|
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
|
||||||
- **Vault repo git objects can be unwritable:** `/home/hermes/wiki/.git` still holds root-owned object fanout dirs/objects (`04`, `42`, `82`, `91`) left by root-run maintenance commits on 2026-07-22. When a new blob or tree hashes into one of those fanouts, `git commit` dies with `insufficient permission for adding an object to repository database` (even though `git add`/`write-tree` look fine). Workaround that works: build the objects in a writable store via `GIT_OBJECT_DIRECTORY=<tmp>` + `GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/hermes/wiki/.git/objects`, then `git pack-objects /home/hermes/wiki/.git/objects/pack/pack-<name>` and move the branch with `git update-ref refs/heads/master <commit>`. Permanent fix is a root `chown -R hermes:hermes /home/hermes/wiki/.git`. Do **not** run overwrite/rename probes against `.git/objects`: on 2026-09-30 one such probe overwrote loose tree `0d23eba37eb65f5e5021f558a370358f1e394c58` (breaking the 2026-07-22 commit `9141a88`); it was reconstructed byte-exactly from its parent tree `d8185c99` plus the two replaced children (`comparisons` tree `822b90a7`, `log.md` blob `043dc1cd`) and reinserted, and `git fsck` is clean again.
|
- **Vault repo git objects can be unwritable:** `/home/hermes/wiki/.git` still holds root-owned object fanout dirs/objects (`04`, `42`, `82`, `91`) left by root-run maintenance commits on 2026-07-22. When a new blob or tree hashes into one of those fanouts, `git commit` dies with `insufficient permission for adding an object to repository database` (even though `git add`/`write-tree` look fine). Workaround that works: build the objects in a writable store via `GIT_OBJECT_DIRECTORY=<tmp>` + `GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/hermes/wiki/.git/objects`, then `git pack-objects /home/hermes/wiki/.git/objects/pack/pack-<name>` and move the branch with `git update-ref refs/heads/master <commit>`. Permanent fix is a root `chown -R hermes:hermes /home/hermes/wiki/.git`. Do **not** run overwrite/rename probes against `.git/objects`: on 2026-09-30 one such probe overwrote loose tree `0d23eba37eb65f5e5021f558a370358f1e394c58` (breaking the 2026-07-22 commit `9141a88`); it was reconstructed byte-exactly from its parent tree `d8185c99` plus the two replaced children (`comparisons` tree `822b90a7`, `log.md` blob `043dc1cd`) and reinserted, and `git fsck` is clean again.
|
||||||
- Browser persistence not yet fully verified
|
- Browser persistence not yet fully verified
|
||||||
|
|||||||
@@ -3,11 +3,11 @@ title: Messaging Integrations
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-07-22
|
updated: 2026-10-01
|
||||||
verified_on: 2026-07-22
|
verified_on: 2026-10-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [messaging, telegram, discord, email, integrations]
|
tags: [messaging, telegram, discord, email, integrations]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Messaging Integrations
|
# Messaging Integrations
|
||||||
@@ -20,6 +20,11 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
|
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721`, `telegram:-1004321904721:notifications`, `telegram:-1004321904721:102`, `telegram:-1004321904721:273`, `telegram:-1004321904721:346`, `telegram:-1004321904721:1030`
|
||||||
- **Notes:** Primary delivery target for scheduled jobs
|
- **Notes:** Primary delivery target for scheduled jobs
|
||||||
|
|
||||||
|
### Delivery target semantics (verified 2026-10-01)
|
||||||
|
- A **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)** — *not* to the primary group. It is not a "send to the main chat" shorthand.
|
||||||
|
- To reach a forum topic you must use the full form **`telegram:<chat_id>:<thread_id>`**, e.g. `telegram:-1004321904721:102`.
|
||||||
|
- Consequence: a job that names a group topic ID but is actually pointed at the DM hard-fails with `configured thread_id <n> … was not found` / `Message thread not found`, and **the message is lost** — it is not silently downgraded to a thread-less send. This caused the 2026-09-30 delivery incident on `fad5a86979d4`, `1b23c042e17d` and `c8d52904e1d6`.
|
||||||
|
|
||||||
## Discord
|
## Discord
|
||||||
- **Status:** enabled, streaming true
|
- **Status:** enabled, streaming true
|
||||||
- **Platform:** `hermes-discord`
|
- **Platform:** `hermes-discord`
|
||||||
@@ -45,7 +50,8 @@ sources: [raw/configs/hermes-config-sanitized.txt]
|
|||||||
- **Home Assistant:** homeassistant-platform enabled
|
- **Home Assistant:** homeassistant-platform enabled
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- **Telegram thread delivery failures on `telegram:1793951355` (2026-09-30):** jobs `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reports `Message thread not found` (`last_status: delivery_failed`). Content generation succeeds — only the thread target is wrong. Fix: recreate the topic, or clear `thread_id` on those jobs. See [[systems/scheduled-tasks]].
|
- **Telegram thread-delivery failures — resolved 2026-09-30.** Jobs `fad5a86979d4` and `1b23c042e17d` reported `configured thread_id 34795 … was not found`, and `c8d52904e1d6` reported `Message thread not found` (`last_status: delivery_failed`). Root cause was the *target*, not the topic: those jobs used a bare `deliver: "telegram"`, which resolves to the home DM where group topic IDs do not exist. `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/scheduled-tasks]].
|
||||||
|
- **Three jobs still deliver to the home DM via a bare `telegram` target** (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Not failing today, but they carry stale `thread_id` errors and sit in the same failure class as the incident above — retarget them to a group topic.
|
||||||
|
|
||||||
## Multi-bot group chat (open question, raised 2026-09-30)
|
## Multi-bot group chat (open question, raised 2026-09-30)
|
||||||
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
|
Anthony asked whether his other Hermes instances (the estate VMs/CTs) could share a group chat.
|
||||||
|
|||||||
+12
-2
@@ -3,8 +3,8 @@ title: Mnemosyne
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-02
|
updated: 2026-10-01
|
||||||
verified_on: 2026-08-02
|
verified_on: 2026-10-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, backend]
|
tags: [memory, mnemosyne, backend]
|
||||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||||
@@ -65,6 +65,12 @@ Primary agent memory store for Hermes.
|
|||||||
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
|
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
|
||||||
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
|
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
|
||||||
|
|
||||||
|
### Gateway notifications were captured as user memories (fixed 2026-09-30)
|
||||||
|
- Hermes's background-process notifications (long-running `terminal` jobs reporting completion) were being written into Mnemosyne as **`[USER] [Anthony Martin]`** memories — i.e. machine messages were stored as if the user had said them, and then surfaced during recall as if they were his words.
|
||||||
|
- **36 such rows** were found and purged on 2026-09-30 (working tier plus the corresponding episodic rows). Recall after the purge returned clean.
|
||||||
|
- Two **canonical facts** were also wrong because of the same class of drift: `hermes_cron_model_default` named the removed provider `custom:omni` instead of `custom:omni-101`, and the Daily Marketing Job Radar's model fact was stale. Both corrected.
|
||||||
|
- **Rule:** treat a recalled memory attributed to a person as untrusted until the attribution makes sense in context. Machine-generated status text should never enter the person-attributed tier.
|
||||||
|
|
||||||
## Known Management Jobs
|
## Known Management Jobs
|
||||||
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||||||
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||||
@@ -72,6 +78,9 @@ Primary agent memory store for Hermes.
|
|||||||
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||||
|
|
||||||
## History and Evidence
|
## History and Evidence
|
||||||
|
- **2026-09-30 memory-hygiene pass:** 36 gateway-notification rows purged from the person-attributed tier; two canonical facts corrected (`hermes_cron_model_default` → `custom:omni-101`, job-radar model). No unresolved conflicts left in the working tier.
|
||||||
|
- **2026-08-19 Mnemosyne Watchdog incident:** Watchdog job (`96eb95ce12f2`) cycled for ~2 hours early morning Aug 19 reporting no consolidation for 36–41.5+ hours (last consolidation Aug 17 07:15). Watchdog stopped after ~2 hours of alerts. See [[systems/scheduled-tasks]] Known Issues section.
|
||||||
|
- **2026-08-18 incident:** Working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39 — ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. Config is clean (native YAML types). MEMDEBUG logging recommended to identify which guard blocks. See [[incidents/2026-08-18-mnemosyne-capture-pipeline]].
|
||||||
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
|
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
|
||||||
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
|
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
|
||||||
@@ -81,4 +90,5 @@ Primary agent memory store for Hermes.
|
|||||||
- [[concepts/agent-memory]]
|
- [[concepts/agent-memory]]
|
||||||
- [[comparisons/memory-backends]]
|
- [[comparisons/memory-backends]]
|
||||||
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||||||
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]]
|
||||||
- [[incidents/2026-07-29-mnemosyne-relay]]
|
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||||
|
|||||||
+36
-12
@@ -3,8 +3,8 @@ title: Model Providers
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-08-20
|
updated: 2026-10-01
|
||||||
verified_on: 2026-08-20
|
verified_on: 2026-10-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [model, provider, inference, routing]
|
tags: [model, provider, inference, routing]
|
||||||
sources: [live-probe, config, web-search]
|
sources: [live-probe, config, web-search]
|
||||||
@@ -16,18 +16,42 @@ sources: [live-probe, config, web-search]
|
|||||||
Inference providers available to Hermes, their routing paths, and notable model updates.
|
Inference providers available to Hermes, their routing paths, and notable model updates.
|
||||||
|
|
||||||
## Current Provider Stack
|
## Current Provider Stack
|
||||||
| Provider | URL | Role |
|
Verified from live `~/.hermes/config.yaml` → `custom_providers`, 2026-10-01.
|
||||||
|---|---|---|
|
|
||||||
| `custom:litellm` | `http://litellm:4000/v1` | Main inference (slower path, 6–26s/call) |
|
| Provider | URL | Models | Role |
|
||||||
| `custom:omni` | `http://omni:20128/v1` | Auxiliary roles (vision, skills, approval, mcp, web_extract, compression) — factory-reset 2026-08-14 |
|
|---|---|---|---|
|
||||||
| `opencode-zen` | (internal) | Free-tier model access for cron jobs |
|
| `custom:omni-101` | `http://100.78.63.20:20128` | 5,384 | **Primary.** OmniRoute on the estate (`omni-101`). 15 of 30 cron jobs reference it |
|
||||||
| `charm-hyper` | `hyper.charm.land/v1` | Coding-focused inference (Charmbracelet) — passthrough models, free tier 100 credits/mo |
|
| `custom:Omniroute:20128` | `http://omniroute:20128/v1` | 5,398 | Secondary OmniRoute host — legacy naming, kept alongside `omni-101` |
|
||||||
| Guanaco | `ollama:11434` | LLM proxy/router on CT205 (Ollama stack) |
|
| `custom:litellm` | `http://litellm:4000/v1` | 1,728 | Fallback inference (slower path, 6–26s/call) |
|
||||||
|
| `custom:command-code` | `https://api.commandcode.ai/provider/v1` | 71 | DeepSeek Flash primary via the `deepseek-flash-mix` combo (added 2026-08-19); also the source of the `MODEL_NOT_IN_PLAN` 403 on `gpt-5.6-terra` |
|
||||||
|
| `custom:guacano` | `http://ollama:11000/v1` | 46 | Guanaco LLM proxy/router on CT205 (Ollama stack). See [[systems/guanaco]] |
|
||||||
|
| `custom:omni` | *(removed)* | — | **No longer configured.** It once served auxiliary roles (vision, skills, approval, mcp, web_extract, compression). Two cron jobs still name it as their provider key — see [[systems/scheduled-tasks]] |
|
||||||
|
|
||||||
|
**Built-in (non-`custom_providers`) routes still in use as fallbacks:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`. `charm-hyper` (Charmbracelet, `hyper.charm.land/v1`, free tier 100 credits/mo) is no longer in the live stack.
|
||||||
|
|
||||||
|
**Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute). **Default model:** `smart-route`.
|
||||||
|
|
||||||
|
Provider names that appear in job configs but not in `custom_providers` do **not** necessarily fail: the default `custom` block resolves to the same base URL, so a stale provider key can go unnoticed. Prefer routing aliases (`smart-route`) over raw upstream model names in pins.
|
||||||
|
|
||||||
## Default Model
|
## Default Model
|
||||||
- `resilient-free` — resolves through OmniRoute (changed from `auto/best-free` on 2026-08-14 after factory reset)
|
- `smart-route` — the live default as of 2026-09-30 (`provider: custom`, `base_url: http://100.78.63.20:20128`)
|
||||||
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to paid `gpt-5.6-sol`
|
- Anthony has explicitly **rejected** `free-lkgp` as a default; stable defaults are `smart-route` / `deepseek-paid-smart` (paid, not free)
|
||||||
- `freellmapi-auto` (LiteLLM path) still available but slower
|
- `resilient-free` was the default after the 2026-08-14 OmniRoute factory reset, but that combo is now in the broken set
|
||||||
|
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to a paid `gpt-5.6-sol`
|
||||||
|
|
||||||
|
## Retired / Unavailable Models (2026-09-25 → 2026-10-01)
|
||||||
|
Pins that fail at request time, with the exact upstream error. Each is an open cron-repin in [[systems/scheduled-tasks]].
|
||||||
|
| Pin | Upstream | Error | Affected job |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `qwen3.5:397b` | `ollama-cloud` | `HTTP 410: qwen3.5:397b was retired at 2026-09-25` | `1a8e90ea2271` (wiki stocktake, streak 7) |
|
||||||
|
| `gpt-5.6-terra` | `command-code` | `403 MODEL_NOT_IN_PLAN: GPT-5.6 Terra available in Pro and above plans` | `dca8482e4f76` (Daily Marketing Job Radar) |
|
||||||
|
| `nvidia/deepseek-ai/deepseek-v4-flash` | `nvidia` via `custom:omni-101` | `HTTP 400: Model 'deepseek-ai/deepseek-v4-flash' is not available in the active live catalog for provider 'nvidia'` | `fad5a86979d4` (Facebook birthday sync) |
|
||||||
|
|
||||||
|
Lesson: a retired or plan-gated **raw model pin never self-recovers** — it fails identically on every run until someone changes the pin. Pin to a routing alias instead.
|
||||||
|
|
||||||
|
## Auxiliary Role Routing
|
||||||
|
- **Vision:** live `vision.model` is `stepfun/step-3.7-flash:free` on provider `nous`. Anthony asked on 2026-09-30 that vision calls use the **local** Ollama on `http://anthonys-mac-mini:11434` (gemma4) going forward — as of 2026-10-01 the config still points at `nous`, so the switch is recorded but not applied.
|
||||||
|
- Auxiliary roles (vision, skills, approval, mcp, web_extract, compression) previously routed through `custom:omni`, which no longer exists.
|
||||||
|
|
||||||
## Notable Model Updates
|
## Notable Model Updates
|
||||||
|
|
||||||
|
|||||||
+23
-18
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-09-30
|
updated: 2026-10-01
|
||||||
verified_on: 2026-09-30
|
verified_on: 2026-10-01
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [cron, scheduled, automation]
|
tags: [cron, scheduled, automation]
|
||||||
sources: [live-cron-list, live-jobs-json]
|
sources: [live-cron-list, live-jobs-json]
|
||||||
@@ -13,10 +13,10 @@ sources: [live-cron-list, live-jobs-json]
|
|||||||
# Scheduled Tasks
|
# Scheduled Tasks
|
||||||
|
|
||||||
## Count
|
## Count
|
||||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-09-30)
|
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-01)
|
||||||
- **Failing:** 4 (`dca8482e4f76` model-plan 403 ×5; `fad5a86979d4` thread; `cbfe434a3b5e` + `1a8e90ea2271` residual pre-fix dispatch error)
|
- **Failing:** 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, streak 5), `1a8e90ea2271` (HTTP 410, model retired upstream, streak 7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, streak 3)
|
||||||
- **Delivery-only failures:** 2 (`c8d52904e1d6`, `1b23c042e17d` — content ran, Telegram thread missing)
|
- **Stale status flag:** 1 — `1b23c042e17d` still reads `delivery_failed` from its pre-fix run. The delivery target was corrected on 2026-09-30 and its next run clears the flag.
|
||||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30)
|
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01)
|
||||||
|
|
||||||
## Job Table
|
## Job Table
|
||||||
|
|
||||||
@@ -27,12 +27,12 @@ sources: [live-cron-list, live-jobs-json]
|
|||||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×5 — see Known Issues |
|
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×5 — see Known Issues |
|
||||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
||||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram | — | — | ⚠ `thread_id 34795` not found → delivered without thread |
|
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30 (was bare `telegram` → home DM); `delivery_failed` flag is stale |
|
||||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | |
|
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | bare `telegram` → home DM; should be retargeted to a group topic |
|
||||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ `thread_id 34795` not found → delivered without thread |
|
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ HTTP 400 ×3 — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; bare `telegram` target should move to a group topic |
|
||||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ Telegram bot token in `github-trending-radar/config.json` revoked (`getMe` → 401); digest builds, push fails |
|
||||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||||
@@ -42,15 +42,15 @@ sources: [live-cron-list, live-jobs-json]
|
|||||||
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||||
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) |
|
||||||
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||||
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram | — | — | ⚠ delivery error "Message thread not found" |
|
| Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` |
|
||||||
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||||
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | bare `telegram` → home DM; carries a stale DM-thread delivery error |
|
||||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | |
|
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | |
|
||||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
|
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×7 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
|
||||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||||
|
|
||||||
## Removed Jobs
|
## Removed Jobs
|
||||||
@@ -70,13 +70,18 @@ sources: [live-cron-list, live-jobs-json]
|
|||||||
## Known Issues
|
## Known Issues
|
||||||
|
|
||||||
### Resolved 2026-09-30
|
### Resolved 2026-09-30
|
||||||
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. 10/10 managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
- **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]].
|
||||||
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". Script restored 2026-09-30 19:17; job now `ok`, streak 0.
|
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||||
|
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0.
|
||||||
|
- **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below).
|
||||||
|
|
||||||
### Open
|
### Open
|
||||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30, streak 5):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change. See [[runbooks/job-radar]].
|
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 30, streak 5):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
||||||
- **Telegram thread delivery failures (`telegram:1793951355`):** `fad5a86979d4` and `1b23c042e17d` report `configured thread_id 34795 … was not found`; `c8d52904e1d6` reports `Message thread not found` (last_status `delivery_failed`). Content runs successfully — only the thread target is wrong. Fix: either recreate the topic or clear `thread_id` from the jobs.
|
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. Until then the daily 03:00 stocktake (skills-index regeneration, freshness audit, Quartz rebuild, Gitea commit) does not run.
|
||||||
- **Residual dispatch errors:** `cbfe434a3b5e` (streak 5) and `1a8e90ea2271` (streak 6) still show the pre-2026-09-30 worker error from runs made before the dependency guard was installed (23:59 AWST and 03:00 AWST respectively). Expected to clear on their next runs — re-check 2026-10-01 03:00.
|
- **Facebook Birthday Weekly Registry Sync — HTTP 400, model missing from catalog (streak 3):** Job `fad5a86979d4` is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. The sibling job `0e753e9720fd` uses the same pin, so this is an upstream catalog/availability issue rather than a per-job config error. Safest durable fix is repinning to a routing alias (`smart-route`) instead of a raw model name.
|
||||||
|
- **GitHub Trending Radar — revoked Telegram bot token:** Job `c0a9fa314542` builds its digest successfully, then fails the Telegram push (`getMe` → 401 Unauthorized). The token lives in `/home/hermes/workspace/github-trending-radar/config.json` and needs replacing from @BotFather. Digest content is recoverable from the script's `--dry-run` output in the interim.
|
||||||
|
- **Stale provider key `custom:omni` (2 jobs):** the provider was removed from `custom_providers` (the live 2026-10-01 set is `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Jobs `e530c89f2be0` and `d40085da631c` still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`.
|
||||||
|
- **Bare `telegram` delivery targets (3 jobs):** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. Not a failure by itself, but it is the same class of misconfiguration that caused the 2026-09-30 thread-delivery incident, and it leaves stale `thread_id` errors on their records. Retarget to `telegram:-1004321904721:<topic>`.
|
||||||
|
|
||||||
### Historical
|
### Historical
|
||||||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||||
|
|||||||
Reference in New Issue
Block a user