Wiki audit 2026-09-30: managed-runtime dependency incident, cron fleet reconciliation, provider/model refresh

This commit is contained in:
Tony0410
2026-09-30 19:38:49 +08:00
parent 389de2119b
commit 58312cb434
8 changed files with 245 additions and 75 deletions
+19 -9
View File
@@ -3,27 +3,37 @@ title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-09-04
review_after: 2026-10-22
updated: 2026-09-30
verified_on: 2026-09-30
review_after: 2026-10-30
confidence: high
tags: [system]
sources: []
sources: [live-config, live-cron-list, live-hermes-version]
---
# Current State
Updated: 2026-09-04
Updated: 2026-09-30
## Recent Changes (2026-09-30)
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
- **Open — Telegram thread delivery failures:** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Content runs; delivery does not.
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
## Recent Changes (2026-09-02)
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
## Attested facts
- **Config version:** 33 — Verified, `~/.hermes/config.yaml`, 2026-07-25
- **Config version:** `_config_version: 46` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously recorded as 33 on 2026-07-25)
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
- **Main provider:** `custom:litellm` → `http://litellm:4000/v1` — Verified, `~/.hermes/config.yaml`, 2026-07-22
- **Default model:** `freellmapi-auto` — Verified, `~/.hermes/config.yaml`, 2026-07-22
- **Fallback/aux provider:** `custom:omni` → `http://omni:20128/v1` — Verified, config, 2026-08-13 (migrated from `custom:omniroute`, CT600)
- **OmniRoute host:** `omni.kangaroo-eel.ts.net` → `100.93.204.4` — Verified, DNS resolution, 2026-08-13
- **Main provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `custom:litellm` → `litellm:4000`, then `custom:omni` → `omni.kangaroo-eel.ts.net`)
- **Default model:** `smart-route` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `freellmapi-auto`)
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` — Verified, config, 2026-09-30
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` — Verified, config, 2026-09-30
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
- **Memory enabled:** true — Verified, config, 2026-07-22
- **Streaming enabled:** true — Verified, config, 2026-07-22