wiki: nightly audit 2026-10-01 — cron fleet re-verified, Telegram delivery root cause, memory hygiene
This commit is contained in:
+13
-4
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-09-30
|
||||
verified_on: 2026-09-30
|
||||
updated: 2026-10-01
|
||||
verified_on: 2026-10-01
|
||||
review_after: 2026-10-30
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,14 +12,23 @@ sources: [live-config, live-cron-list, live-hermes-version]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-09-30
|
||||
Updated: 2026-10-01
|
||||
|
||||
## Recent Changes (2026-10-01)
|
||||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **4 → 3**: `cbfe434a3b5e` returned to `ok` as predicted on 2026-09-30. The residual `1a8e90ea2271` dispatch error also cleared — its current failure is a different, upstream problem (below).
|
||||
- **Telegram delivery failures root-caused and fixed:** the target, not the topic, was wrong — a bare `deliver: "telegram"` resolves to the **home DM (`1793951355`)**, where group topic IDs cannot exist, so sends hard-failed and the message was lost. `1b23c042e17d` and `c8d52904e1d6` retargeted to `telegram:-1004321904721:102` and verified. Three jobs still carry bare `telegram` targets (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). See [[systems/messaging-integrations]].
|
||||
- **Three jobs pinned to models that no longer exist upstream — cannot self-recover:** `1a8e90ea2271` → `qwen3.5:397b` retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`, streak 5); `fad5a86979d4` → `nvidia/…/deepseek-v4-flash` absent from the live catalog (HTTP 400, streak 3). Repin to the `smart-route` alias. See [[systems/model-providers]].
|
||||
- **GitHub Trending Radar — revoked Telegram bot token:** `c0a9fa314542` builds its digest then fails the push (`getMe` → 401). Token in `/home/hermes/workspace/github-trending-radar/config.json` needs replacing from @BotFather.
|
||||
- **Memory hygiene pass (2026-09-30):** 36 gateway background-process notifications had been written into Mnemosyne as `[USER] [Anthony Martin]` memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts `hermes_cron_model_default` and the job-radar model corrected. See [[systems/mnemosyne]].
|
||||
- **Provider stack rebuilt from live config:** 5 custom providers (`omni-101`, `Omniroute:20128`, `litellm`, `command-code`, `guacano`). The provider key `custom:omni` **no longer exists**; two cron jobs (`e530c89f2be0`, `d40085da631c`) still name it but have not failed, because the default `custom` block resolves to the same base URL.
|
||||
- **Vision routing — request recorded, not applied:** Anthony asked on 2026-09-30 that vision calls use the local Ollama at `http://anthonys-mac-mini:11434` (gemma4). Live config still points at `nous / stepfun/step-3.7-flash:free` as at 2026-10-01.
|
||||
|
||||
## Recent Changes (2026-09-30)
|
||||
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
|
||||
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
|
||||
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
|
||||
- **Open — Telegram thread delivery failures:** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Content runs; delivery does not.
|
||||
- **Open — Telegram thread delivery failures (root cause found 2026-10-01):** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Cause was the delivery target, not the topic — a bare `telegram` resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
|
||||
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
|
||||
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user