Files
agent-estate-wiki/systems/hermes-agent.md
T
Tony0410 f880ba981d audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
2026-10-28 00:20:59 +08:00

16 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Hermes Agent system active 2026-07-22 2026-10-28 2026-10-28 high
system
hermes
agent
llm
live-hermes-version
live-config
live-skill-inventory

Hermes Agent

Purpose

Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.

Current Status

  • Version: 0.22.3 — install stamp 5b147ab036d6d19131bff8d20701b5001eab844a, built 2026-10-21T14:57:38Z; git-describe token v0.21.5+15956.g5b147ab (detached HEAD). Last update: 2026-10-21 22:47:25 → 22:57:38 AWST via hermes update (action 9eed3a99e799347c691ea52a72e122a1, receipt update_20261021_144738_866674_82212ebd…): stages plan → snapshot (skipped) → apply (mode git) → deps → build, all success except the skip. The pre-update tree read 0.22.2 / d923a5eb487a17c0a68801bba9ef0f56c81040a8 — an intermediate rebuild never recorded on this page, so the previously-noted v0.21.5+6169.gab86a20 (2026-10-04) is at least two builds behind. Runtimes in the plan: gateway (systemd) and dashboard (manual-serve, restarted by argv). Post-restart state: gateway pid 868087, phase: running, api_server: connected (heartbeat 2026-10-22 00:13 AWST); dashboard pid 869852 on :9119.
  • Install method: git
  • Python: managed runtime 3.14 (legacy venv/ on 3.11.15)
  • Upstream status: advanced 2026-10-21 to commit 5b147ab036 (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ That update ran with its own safety net switched off: the snapshot stage is recorded as skipped, reason disabled by updates.pre_update_backup (mode: off) — the same pre_update_backup: false recorded in infrastructure/backups, while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (hermes sessions optimize-storage — "on the compact layout from an earlier optimization, but a later release changed it"), and → Preparing pinned cua-driver (Computer Use)..., whose completion line is absent from the log — status unverified. The .restart_pending.json request (via_service: true, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.
  • Install directory: /home/hermes/.hermes/hermes-agent
  • Managed runtimes: 3 uv-managed venvs under ~/.hermes/installs/019d0d114d22b7e6/environments/<hash>/venv (was 10 — the 2026-10-03 rebuild pruned superseded environments and created two new ones at 09:59/10:00; the third survives from 2026-09-28). hermes-runtime-deps-guard.sh runs every 15 min via mnemosyne-watchdog.py and now asserts dotenv + mnemosyne in the installer python and every managed venv, plus fastembed + FASTEMBED_CACHE_PATH + the local model dir in the bootstrap-selected venv — see incidents/2026-09-28-hermes-managed-runtime-deps and incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
  • Community resources: Hermes Registry — third-party plugins, skills, and extensions
  • Binary: /home/hermes/.local/bin/hermes
  • Config: ~/.hermes/config.yaml
  • Profile: default (top-level config ~/.hermes/config.yaml; the only named sub-profile on this host is ops at ~/.hermes/profiles/ops/)

Location

  • Host: Local Linux host (no containerized Hermes runtime confirmed)
  • Filesystem root: ~/.hermes/
  • Working directory: /home/hermes/workspace

Providers & Routing

  • Default provider: custom → http://100.78.63.20:20128 (OmniRoute) — verified 2026-09-30
  • Default model: smart-route — verified 2026-09-30
  • History: as at 2026-07-22 this was custom:litellm → litellm:4000 with model freellmapi-auto
  • Provider model count (2026-10-01): 5 custom providers — omni-101 5,384, Omniroute:20128 5,398, litellm 1,728, command-code 71, guacano 46
  • MCP servers: 5 configured (Not human search, asana, cua-driver, hound, mcp-portal) — verified 2026-10-01
  • Plugins enabled: 11 — verified 2026-10-01
  • free-stack combo (OmniRoute): Nemotron 3 Ultra 550B configured with max_tokens: 8192, reasoning_effort: medium, chat_template_kwargs.force_nonempty_content: true (see omniroute)
  • See model-providers for full breakdown.

Memory

  • Backend: Mnemosyne (local native)
  • Memory enabled: true
  • Char limit: 5000
  • User profile enabled: true
  • User char limit: 3000
  • Flush every: 6 turns
  • Auto sleep: true
  • Default scope: session
  • Vector type: int8
  • Shared surface: data/shared/mnemosyne.db
  • See memory-backend for details.

Browser

  • Cloud provider: Camoufox
  • Managed persistence: true
  • Inactivity timeout: 120 s
  • Allow private URLs: true
  • cua-driver: v0.9.0 at /home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver
  • DISPLAY: :102 (Xvfb/X11VNC/fluxbox confirmed working per session notes)
  • See browser-backend for details.

Terminal

  • Backend: local
  • Working directory: /home/hermes/workspace
  • Timeout: 180 s
  • Home mode: auto
  • Container CPU: 1
  • Persistent shell: true
  • See terminal-backend for details.

Skills

  • Total installed: 344 SKILL.md files across 72 top-level skill directories (live count 2026-10-03; re-confirmed by the manual stocktake the same day — the last generated skills-index also reads 344 / 70 categories)
  • Newest addition: fleet-wide-failure-triage (devops) — surfaced by the 2026-10-03 regeneration
  • Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
  • See skills-index for categorized list (auto-regenerated daily — but the generating job is currently blocked, so it is being refreshed by hand; see Known Issues).

Multi-Profile Multiplexer

The default gateway runs as a profile multiplexer (gateway.multiplex_profiles: true), serving both the default and ops profiles through a single gateway process with one Telegram connection.

Profiles

Profile Chat Purpose
default -1004321904721 (Rhino chat) Main agent
ops -1003914987043 (Elephant ops), -1003932503629 (OPS Tember rescue) Operations/infra agent

How It Works

  • One gateway, one Telegram bot token — shared across profiles
  • profile_routes maps chat_id → profile at the gateway level
  • allowed_chats on both profiles must include ALL chats that either profile should hear, because the default profile's adapter processes all inbound messages before profile_routes stamps source.profile
  • The ops profile does NOT need platforms.telegram.enabled — the default adapter handles it

Critical Rule: allowed_chats Is Pre-Route

The allowed_chats gate runs on the default profile's adapter before profile_routes matching. If a chat_id is only in the ops profile's allowed_chats but not the default's, messages from that chat are silently dropped before routing ever fires.

Fix applied 2026-07-28: Added both ops group IDs to the default profile's allowed_chats and group_allowed_chats (comma-separated). See multiplexer-setup.

Messaging Integrations

  • Telegram: enabled, allowed_chats: -1004321904721,-1003914987043,-1003932503629, rich messages
  • Discord: enabled, streaming true
  • Slack: enabled
  • Email: ⚠ partially connected — personal Gmail works; the work Outlook account is unreachable (tenant allows only XOAuth2, the account uses password auth). See incidents/2026-08-26-work-outlook-basic-auth-disabled.
  • Mattermost: connected; gateway warning on 2026-08-15 was caused by a disabled system-level hermes-gateway.service unit, not a live duplicate gateway. Only the user-level service is active.
  • ntfy: connected
  • Google Chat: available (hermes-google_chat)
  • QQ: available (hermes-qqbot)
  • Yuanbao: available
  • See messaging-integrations for details.

Scheduled Tasks

  • Cron jobs: 30 total (27 enabled, 3 disabled) — verified 2026-10-03
  • Failing: 6 — dca8482e4f76 (403 plan-gated), 1a8e90ea2271 (HTTP 410, retired model), fad5a86979d4 (HTTP 400), 0e753e9720fd (HTTP 400), 06a0a1c8704f and 1b23c042e17d (HTTP 401 — the cliprox/terra provider connection auth expired, now confirmed on 2 of the 3 jobs sharing that pin)
  • Delivery targets: Telegram (primary), Discord, origin, local
  • Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
  • See scheduled-tasks for full table.

Dependencies

  • LiteLLM proxy at litellm:4000
  • OmniRoute at omniroute:20128
  • cua-driver binary at path above
  • Xvfb + X11VNC + fluxbox for headed browser
  • Telegram bot token, Discord token, etc. (stored outside wiki)

Known Issues

  • Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • Vector recall silently ran keyword-only until 2026-10-03 (repaired). fastembed was missing from every uv-managed runtime and FASTEMBED_CACHE_PATH was unset, so the dense leg of mnemosyne_recall returned no candidates and recall degraded to FTS5 — with no error raised. Repaired 2026-10-03 and the guard extended (above). Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). See incidents/2026-10-03-mnemosyne-vector-recall-fts-only.
  • Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault: 1a8e90ea2271 → qwen3.5:397b retired 2026-09-25 (HTTP 410); dca8482e4f76 → gpt-5.6-terra plan-gated (403 MODEL_NOT_IN_PLAN); fad5a86979d4 + 0e753e9720fd → deepseek-v4-flash absent from the NVIDIA live catalog (HTTP 400); and 06a0a1c8704f + 1b23c042e17d → the cliprox/terra provider connection auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the smart-route alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See systems/scheduled-tasks, systems/model-providers and incidents/2026-10-01-cliproxy-connection-auth-expired.
  • The daily skills stock-take is not running. Job 1a8e90ea2271 is blocked on the retired model above, so skills-index and the freshness audit are only refreshed when wiki-maintain.sh is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
  • Telegram thread delivery failures — root-caused and fixed 2026-09-30. A bare deliver: "telegram" resolves to the home DM (1793951355), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to telegram:-1004321904721:102 and verified. Three jobs still use bare telegram targets. See systems/messaging-integrations.
  • Two wiki trees drift: /home/hermes/workspace/agent-estate-wiki (Gitea + Quartz source, pages at root) and /home/hermes/wiki/agent-estate-wiki (Obsidian/Ignis vault, pages under wiki/). They are separate git repos with different history. See current-state.
  • Vault repo git objects can be unwritable: /home/hermes/wiki/.git still holds root-owned object fanout dirs/objects (04, 42, 82, 91) left by root-run maintenance commits on 2026-07-22. When a new blob or tree hashes into one of those fanouts, git commit dies with insufficient permission for adding an object to repository database (even though git add/write-tree look fine). Workaround that works: build the objects in a writable store via GIT_OBJECT_DIRECTORY=<tmp> + GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/hermes/wiki/.git/objects, then git pack-objects /home/hermes/wiki/.git/objects/pack/pack-<name> and move the branch with git update-ref refs/heads/master <commit>. Permanent fix is a root chown -R hermes:hermes /home/hermes/wiki/.git. Do not run overwrite/rename probes against .git/objects: on 2026-09-30 one such probe overwrote loose tree 0d23eba37eb65f5e5021f558a370358f1e394c58 (breaking the 2026-07-22 commit 9141a88); it was reconstructed byte-exactly from its parent tree d8185c99 plus the two replaced children (comparisons tree 822b90a7, log.md blob 043dc1cd) and reinserted, and git fsck is clean again.
  • Browser persistence not yet fully verified

2026-10-10 - gateway runtime env, TTS, Paseo

  • Gateway process shape (verified 2026-10-11 00:20): PID 230494, started 2026-10-10 21:24:44, launched as /home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/bin/python3 -I -c '<bootstrap>' gateway run (RSS ~371 MB). activate_dependencies() replaces sys.path with a single committed uv environment (~/.hermes/installs/<id>/environments/795020dbf24343cd9c52df24723dca83/venv), and the user site directory is invisible to it - packages the gateway needs must be installed into that env. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • Mnemosyne provider healthy again: Memory provider 'mnemosyne' registered (40 tools) + activated from 2026-10-10 22:01:19 onward (repeated 23:59 / 00:00 / 00:02); hermes memory status reports available.
  • TTS: ElevenLabs is the configured provider; its package was missing from the live managed env and was reinstalled 2026-10-10. voice.auto_tts: false - auto-speak is off by design, so TTS output appears only when explicitly requested.
    • ⚠ Superseded — live config checked 2026-10-28 (change date not established): tts.provider is openai pointed at the Mac mini speech server — tts.providers.openai.base_url = https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech, model: phonon, voice: alloy. Live voice (as opposed to one-shot TTS) runs over voice.speech_server.url = wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/, with wake_word.enabled: true. A command-type provider ollama_local (~/.hermes/scripts/ollama-tts.sh) remains configured as an alternative. tts.elevenlabs (eleven_flash_v2_5) and tts.gemini (gemini-2.5-flash-preview-tts, voice Charon, persona file ~/.hermes/hermes-tts-persona.txt) are still present as provider blocks. voice.auto_tts is still false. See systems/conduit for the voice path.
  • Mattermost platform unreachable since 2026-10-10; the gateway keeps retrying on a 5-minute cadence - systems/messaging-integrations.
  • Paseo, the coding-agent orchestrator on 0.0.0.0:6767, is documented at systems/paseo.

2026-10-26 00:1x AWST — gateway restart record and the dead Nous credential

  • Three gateway starts on 2026-10-25: 00:32:20, 18:30:04, 19:27:13 AWST (after 2026-10-24 12:24:39). The 18:29:22 shutdown carried signal=SIGTERM under_systemd=yes, loadavg_1m=3.79 — a planned stop, not a crash; 19:26:16 logged "Stopping gateway for restart..." (the plugin self-update path). Current gateway pid 2476414, up since 19:26:33 (≈4 h 45 m at audit time).
  • Consequences of the 19:27 start: the Conduit dashboard gateway restarted at 18:42:33 and so picked up the plugin's upstream commits → systems/conduit; the Mattermost reconnect attempt counter reset to 54 → systems/messaging-integrations; the invalid_grant spam stopped.
  • ⚠ The Nous Portal credential is dead — Nous Portal runtime credentials unavailable: Invalid refresh token. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04, agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 start. Context compression was unavailable for most of 10-25 and contributed to the digest runaway. → incidents/2026-10-22-nous-portal-refresh-token-invalid
  • Restart hygiene: a gateway restart resets in-process counters (Mattermost retries, cron admission bookkeeping). Never read a counter as an outage duration.