Files
agent-estate-wiki/systems/hermes-agent.md
T

256 lines
23 KiB
Markdown

---
title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-11-08
verified_on: 2026-11-08
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
---
# Hermes Agent
## Purpose
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
## Current Status
- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
- **Managed runtimes:** 3 uv-managed venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv` (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). `hermes-runtime-deps-guard.sh` runs every 15 min via `mnemosyne-watchdog.py` and now asserts `dotenv` + `mnemosyne` in the installer python *and every managed venv*, **plus** `fastembed` + `FASTEMBED_CACHE_PATH` + the local model dir in the bootstrap-selected venv — see [[incidents/2026-09-28-hermes-managed-runtime-deps]] and [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
- **Binary:** `/home/hermes/.local/bin/hermes`
- **Config:** `~/.hermes/config.yaml`
- **Profile:** `default` (top-level config `~/.hermes/config.yaml`; the only named sub-profile on this host is `ops` at `~/.hermes/profiles/ops/`)
## Location
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
- **Filesystem root:** `~/.hermes/`
- **Working directory:** `/home/hermes/workspace`
## Providers & Routing
- **Default provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — verified 2026-09-30
- **Default model:** `smart-route` — verified 2026-09-30
- **History:** as at 2026-07-22 this was `custom:litellm` → `litellm:4000` with model `freellmapi-auto`
- **Provider model count (2026-10-01):** 5 custom providers — `omni-101` 5,384, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46
- **MCP servers:** 5 configured (`Not human search`, `asana`, `cua-driver`, `hound`, `mcp-portal`) — verified 2026-10-01
⚠ **Auth state re-verified 2026-11-01: `asana` is enabled but has never been authenticated.** `hermes mcp list` shows it ✓ enabled (`https://mcp.asana.com/v2/mcp`, `auth: oauth`), but `~/.hermes/mcp-tokens/` holds **only** `asana.client.json` (319 B, mtime 2026-10-26 01:07) — no token file — so every attempt logs `MCP OAuth setup failed for 'asana': … non-interactive environment and no cached tokens found. Run \`hermes mcp login asana\` interactively first`. The other four servers are unaffected → [[incidents/2026-10-31-asana-access-failures]]
- **Plugins enabled:** 11 — verified 2026-10-01
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
- See [[model-providers]] for full breakdown.
## Memory
- **Backend:** Mnemosyne (local native)
- **Memory enabled:** true
- **Char limit:** 5000
- **User profile enabled:** true
- **User char limit:** 3000
- **Flush every:** 6 turns
- **Auto sleep:** true
- **Default scope:** session
- **Vector type:** int8
- **Shared surface:** `data/shared/mnemosyne.db`
- See [[memory-backend]] for details.
## Browser
- **Cloud provider:** Camoufox
- **Managed persistence:** true
- **Inactivity timeout:** 120 s
- **Allow private URLs:** true
- **cua-driver:** v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
- **DISPLAY:** `:102` (Xvfb/X11VNC/fluxbox confirmed working per session notes)
- See [[browser-backend]] for details.
## Terminal
- **Backend:** local
- **Working directory:** `/home/hermes/workspace`
- **Timeout:** 180 s
- **Home mode:** auto
- **Container CPU:** 1
- **Persistent shell:** true
- See [[terminal-backend]] for details.
## Skills
- **Total installed:** 344 `SKILL.md` files across 72 top-level skill directories (live count 2026-10-03; re-confirmed by the manual stocktake the same day — the last generated [[skills-index]] also reads 344 / 70 categories)
- Newest addition: `fleet-wide-failure-triage` (devops) — surfaced by the 2026-10-03 regeneration
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
- See [[skills-index]] for categorized list (auto-regenerated daily — but the generating job is currently blocked, so it is being refreshed by hand; see Known Issues).
## Multi-Profile Multiplexer
The default gateway runs as a **profile multiplexer** (`gateway.multiplex_profiles: true`), serving both the `default` and `ops` profiles through a single gateway process with one Telegram connection.
### Profiles
| Profile | Chat | Purpose |
|---------|------|---------|
| default | `-1004321904721` (Rhino chat) | Main agent |
| ops | `-1003914987043` (Elephant ops), `-1003932503629` (OPS Tember rescue) | Operations/infra agent |
### How It Works
- **One gateway, one Telegram bot token** — shared across profiles
- **`profile_routes`** maps chat_id → profile at the gateway level
- **`allowed_chats` on both profiles** must include ALL chats that either profile should hear, because the default profile's adapter processes all inbound messages before `profile_routes` stamps `source.profile`
- The ops profile does NOT need `platforms.telegram.enabled` — the default adapter handles it
### Critical Rule: allowed_chats Is Pre-Route
The `allowed_chats` gate runs on the **default profile's adapter** before `profile_routes` matching. If a chat_id is only in the ops profile's `allowed_chats` but not the default's, messages from that chat are silently dropped before routing ever fires.
**Fix applied 2026-07-28:** Added both ops group IDs to the default profile's `allowed_chats` and `group_allowed_chats` (comma-separated). See [[multiplexer-setup]].
## Messaging Integrations
- **Telegram:** enabled, `allowed_chats: -1004321904721,-1003914987043,-1003932503629`, rich messages
- **Discord:** enabled, streaming true
- **Slack:** enabled
- **Email:** ⚠ partially connected — personal Gmail works; the `work` Outlook account is unreachable (tenant allows only `XOAuth2`, the account uses password auth). See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
- **Mattermost:** connected; gateway warning on 2026-08-15 was caused by a disabled system-level `hermes-gateway.service` unit, not a live duplicate gateway. Only the user-level service is active.
- **ntfy:** connected
- **Google Chat:** available (`hermes-google_chat`)
- **QQ:** available (`hermes-qqbot`)
- **Yuanbao:** available
- See [[messaging-integrations]] for details.
## Scheduled Tasks
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-10-03
- **Failing: 6** — `dca8482e4f76` (403 plan-gated), `1a8e90ea2271` (HTTP 410, retired model), `fad5a86979d4` (HTTP 400), `0e753e9720fd` (HTTP 400), `06a0a1c8704f` and `1b23c042e17d` (HTTP 401 — the `cliprox/terra` provider connection auth expired, now confirmed on 2 of the 3 jobs sharing that pin)
- Delivery targets: Telegram (primary), Discord, origin, local
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
- See [[scheduled-tasks]] for full table.
## Dependencies
- LiteLLM proxy at `litellm:4000`
- OmniRoute at `omniroute:20128`
- cua-driver binary at path above
- Xvfb + X11VNC + fluxbox for headed browser
- Telegram bot token, Discord token, etc. (stored outside wiki)
## Known Issues
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Vector recall silently ran keyword-only until 2026-10-03 (repaired).** `fastembed` was missing from every uv-managed runtime and `FASTEMBED_CACHE_PATH` was unset, so the dense leg of `mnemosyne_recall` returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]].
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
- **Vault repo git objects can be unwritable:** `/home/hermes/wiki/.git` still holds root-owned object fanout dirs/objects (`04`, `42`, `82`, `91`) left by root-run maintenance commits on 2026-07-22. When a new blob or tree hashes into one of those fanouts, `git commit` dies with `insufficient permission for adding an object to repository database` (even though `git add`/`write-tree` look fine). Workaround that works: build the objects in a writable store via `GIT_OBJECT_DIRECTORY=<tmp>` + `GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/hermes/wiki/.git/objects`, then `git pack-objects /home/hermes/wiki/.git/objects/pack/pack-<name>` and move the branch with `git update-ref refs/heads/master <commit>`. Permanent fix is a root `chown -R hermes:hermes /home/hermes/wiki/.git`. Do **not** run overwrite/rename probes against `.git/objects`: on 2026-09-30 one such probe overwrote loose tree `0d23eba37eb65f5e5021f558a370358f1e394c58` (breaking the 2026-07-22 commit `9141a88`); it was reconstructed byte-exactly from its parent tree `d8185c99` plus the two replaced children (`comparisons` tree `822b90a7`, `log.md` blob `043dc1cd`) and reinserted, and `git fsck` is clean again.
- Browser persistence not yet fully verified
## Related
- [[model-providers]]
- [[memory-backend]]
- [[browser-backend]]
- [[terminal-backend]]
- [[skills-index]]
- [[messaging-integrations]]
- [[scheduled-tasks]]
- [[current-state]]
---
## 2026-10-10 - gateway runtime env, TTS, Paseo
- **Gateway process shape (verified 2026-10-11 00:20):** PID 230494, started 2026-10-10 21:24:44,
launched as `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run`
(RSS ~371 MB). `activate_dependencies()` replaces `sys.path` with a single committed uv
environment (`~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv`), and the
user site directory is invisible to it - packages the gateway needs must be installed **into that
env**. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Mnemosyne provider healthy again:** `Memory provider 'mnemosyne' registered (40 tools)` +
`activated` from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); `hermes memory
status` reports available.
- **TTS:** ElevenLabs is the configured provider; its package was missing from the live managed env
and was reinstalled 2026-10-10. `voice.auto_tts: false` - auto-speak is off by design, so TTS
output appears only when explicitly requested.
- ⚠ **Superseded twice — live config checked 2026-10-30:** `tts.provider` is now **`ollama_local`**
(`~/.hermes/scripts/ollama-tts.sh`, model `phonon`) and `stt.provider` is now **`local`** (faster-whisper
`base`); `voice.speech_server.url` is now `wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/realtime`
(the bare `/v1/` answers **403**, and the plugin dials the value verbatim). See the 2026-10-30 section below.
- ⚠ **Superseded — live config checked 2026-10-28** (change date not established): `tts.provider` is
**`openai`** pointed at the **Mac mini speech server** — `tts.providers.openai.base_url` =
`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, `model: phonon`, `voice: alloy`. Live voice
(as opposed to one-shot TTS) runs over `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`, with `wake_word.enabled: true`. A `command`-type
provider `ollama_local` (`~/.hermes/scripts/ollama-tts.sh`) remains configured as an alternative.
`tts.elevenlabs` (`eleven_flash_v2_5`) and `tts.gemini` (`gemini-2.5-flash-preview-tts`, voice `Charon`,
persona file `~/.hermes/hermes-tts-persona.txt`) are still present as provider blocks. `voice.auto_tts`
is still `false`. See [[systems/conduit]] for the voice path.
- **Mattermost platform unreachable** since 2026-10-10; the gateway keeps retrying on a 5-minute
cadence - [[systems/messaging-integrations]].
- **Paseo**, the coding-agent orchestrator on `0.0.0.0:6767`, is documented at [[systems/paseo]].
## 2026-10-30 00:0x AWST — the Conduit voice path: a truncated URL, a 12-hour session TTL, and two dead providers
Anthony asked why Conduit voice calls never got an answer. **Three faults sat in series** — each one masked the next —
and all three were fixed on 2026-10-29 between 08:02 and 08:4x AWST:
| Fault | Was | Symptom | Now |
|---|---|---|---|
| `voice.speech_server.url` | `wss://…:8443/v1/` | dialled **verbatim** → HTTP **403**, reported as *"the speech server refused the key"* | `wss://…:8443/v1/realtime` |
| `dashboard.basic_auth.session_ttl_seconds` | `43200` (12 h default) | expired phone session → **WS 4401 "Dashboard sign-in required"**, no Hermes turn, no error on the device | `2592000` (30 d, matched to the refresh-token life) |
| `tts.provider` / `stt.provider` | `openai` / `groq` | Mac `/v1/speech` **502** / **403 Cloudflare 1010** | `ollama_local` / `local` |
- **Why it stayed invisible:** the Mac recorded **zero connections**, `conduit-calls.json → history` was empty, and each
attempt logged a reason string pointing at the *next* fault — so one broken chain looked like three unrelated problems.
`last_request_at` was `null`: no call had ever reached Hermes' turn logic.
- **The 12 h TTL hid behind the others.** The dashboard had stopped issuing long-enough sessions, and the iPhone only
re-authenticated when the dashboard happened to prompt it — an audit-log entry shows a re-auth at **08:00:59**, minutes
*after* a silent call.
- **Two TTL keys exist.** `dashboard.basic_auth.session_ttl_seconds` was raised; the similarly named
`dashboard.session_ttl_seconds` **still reads 43200**. Check **both** on any future 4401.
- **Groq was blocked, not mis-keyed** — it answers 403/1010 with a bogus key too, so this was Cloudflare bot-blocking
rather than a credential problem.
- **Verified for real:** the whole path was driven (mint HMAC dashboard session → WS ticket → let the plugin dial the Mac),
then two live calls landed at **2026-10-29 08:38:04** and **08:49:36** — the first entries ever in
`conduit-calls.json → history`. `ollama_local` produced **6.94 s** of real 64 kbps speech; `local` STT returned
`success: True`.
- Backups: `config.yaml.bak.url-081525`, `config.yaml.bak.ttl-20261029080258`, `config.yaml.bak.stt-084536`; the dashboard
restarted **08:03:26** → [[incidents/2026-10-29-conduit-voice-silence]] · [[systems/conduit]].
## 2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential
- **Three gateway starts on 2026-10-25**: `00:32:20`, `18:30:04`, `19:27:13` AWST (after `2026-10-24 12:24:39`).
The `18:29:22` shutdown carried `signal=SIGTERM under_systemd=yes`, `loadavg_1m=3.79` — a planned stop, not a
crash; `19:26:16` logged *"Stopping gateway for restart..."* (the plugin self-update path). Current gateway
pid **2476414**, up since **19:26:33** (≈4 h 45 m at audit time).
- **Consequences of the 19:27 start:** the Conduit dashboard gateway restarted at 18:42:33 and so picked up the
plugin's upstream commits → [[systems/conduit]]; the Mattermost reconnect attempt counter reset to 54
→ [[systems/messaging-integrations]]; the `invalid_grant` spam stopped.
- ⚠ **The Nous Portal credential is dead** — `Nous Portal runtime credentials unavailable: Invalid refresh token`.
Gateway-log counts: **1 line 2026-10-22, 3 on 10-24, 20 on 10-25** (last 14:53:04,
`agent.context_compressor: Failed to generate context summary`), **0 since the 19:27:13 start**. Context
compression was unavailable for most of 10-25 and contributed to the digest runaway.
→ [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
- **Restart hygiene:** a gateway restart resets in-process counters (Mattermost retries, cron admission
bookkeeping). Never read a counter as an outage duration.
## 2026-11-01 00:2x AWST — gateway reloads on 10-31 (one force-drained, one clean) and the Asana MCP auth gap
- **Two gateway starts on 2026-10-31, and the first one killed a live cron run.**
- **18:06:41** — `SIGUSR1 received (systemctl reload / hermes gateway restart)`: *"Restart requested with
2 active work unit(s); deferring stop() until they finish (cap=1800s)"*. The countdown ran every 30 s
from **1770 s** to **988 s** (18:20:13) against a single `managed` unit (`pid 397`); a **second
`SIGUSR1`** landed at **18:11:23**. The cap expired at **18:37:38** → *"Skipping `.clean_shutdown`
marker — drain timed out with interrupted agents"*. New gateway **18:38:43**. The 12:01:06 slot of
`ea79a2107e36` (Mnemosyne Health Monitor) closed **`failed`** at **18:39:15** on *"lost its durable
fire claim ownership"* after **6 h 38 m** → [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].
- **19:23:39 → 19:24:40** — a second restart, **clean**: `drain took 0.00s, timed_out=False`,
*"Previous gateway exited cleanly — skipping session suspension"*. Current gateway **PID 175269**, active
since **19:23:56** (4 h 43 m at the audit probe). Nothing failed after 19:15:21.
- **Two Telegram adapter rebuilds, both self-recovered:** `05:03:42` and `09:12:11` —
`Fatal telegram adapter error (telegram_network_error): Telegram updater.stop() did not finish before the
network-recovery deadline; rebuilding the adapter instead of reusing an Updater whose lifecycle lock may
still be held. (likely CLOSE-WAIT socket)`.
- **Plugin update machinery was active minutes before the reload** (`.update-<id>/plugin` capability checks
18:05:41 and 18:19:53, `plugin scan passed` 18:20:22) and the 04:56:54 log line advertised updates for
`claude-subscription-directsdk-experimental`, `conduit_push` and `homeassistant`. **Agent version
unchanged** — `hermes-agent` HEAD is still `bf0203a7` (2026-10-22).
- **MCP inventory unchanged at 5 servers, but `asana` has never authenticated** (see the MCP line above and
[[incidents/2026-10-31-asana-access-failures]]): **7 728** `MCP OAuth setup failed for 'asana'` log lines in
the retained window (≥2026-10-21), **1 156** of them on 10-31.