89 lines
6.2 KiB
Markdown
89 lines
6.2 KiB
Markdown
---
|
|
title: Hermes Production
|
|
type: system
|
|
status: active
|
|
created: 2026-08-15
|
|
updated: 2026-11-08
|
|
verified_on: 2026-11-08
|
|
confidence: high
|
|
tags: [hermes, production, ct460, agent]
|
|
sources: [live-probe, config-inspection]
|
|
---
|
|
|
|
# Hermes Production
|
|
|
|
## Purpose
|
|
|
|
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. This is the production instance — changes here affect live workflows.
|
|
|
|
## Location
|
|
|
|
- **Container:** LXC CT460 (`hermes`)
|
|
- **Tailscale IP:** `100.118.5.51` (⚠ `100.96.100.82` is the **PVE hypervisor** `pve`, not CT460 — corrected 2026-10-26)
|
|
- **Hostname:** `hermes`
|
|
- **Ports:** 9119 (gateway/API), 8085 (webhook) — ⚠ the old **8082** claim was dead: no 8082 listener on CT460 as of 2026-10-26
|
|
- **Memory:** 14GB (upgraded from 5096MB on 2026-08-07)
|
|
|
|
## Current State
|
|
|
|
- **Status:** Active
|
|
- **Hermes version:** ⚠ **`v0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s), `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`**; local version `0.22.5` → `0.23.0`. ⚠ The **`snapshot` stage was skipped again** (`disabled by updates.pre_update_backup (mode: off)`) → **no rollback point**. Restarts: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`); the gateway went round **again 11:11:44 → 11:11:55** (live process `pid 3109529`). **Live-probed 2026-11-08 00:4x AWST.** Previous: `v0.22.3` (2026.9.24), local `5b147ab036d6` — updated 2026-10-21 22:47→22:57 AWST.
|
|
- **Cron ledger 2026-11-07:** 163 rows — **159 `completed` / 3 `failed` / 1 `running`** (that *running* row is this nightly audit). Two of the three failures are one event: the **05:01:00 gateway restart drain** orphaned *Mnemosyne Health Monitor* (`ea79a2107e36`, 00:00:42 → 05:02:22) and *wiki-stocktake-maintenance* (`1a8e90ea2271`, 03:00:31 → 05:02:22) — both closed `failed` on `lost its durable fire claim ownership`. The third (`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog firing 19 s later on the first failure's traceback. → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
|
|
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
|
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
|
- **Config:** `~/.hermes/config.yaml`
|
|
- **Profile:** `default` — top-level config; **no `profiles/default/` directory exists** (`ops` at `~/.hermes/profiles/ops/` is the only named sub-profile)
|
|
- **Soul:** Rhino — trickster-mentor in a green jacket 🎩
|
|
|
|
## Profiles
|
|
|
|
| Profile | Telegram Chats | Purpose |
|
|
|---------|---------------|---------|
|
|
| default | `-1004321904721` (Rhino chat) | Main agent |
|
|
| ops | `-1003914987043` (Elephant ops), `-1003932503629` (OPS Tember rescue) | Operations/infra agent (Tusk) |
|
|
|
|
## Providers & Routing
|
|
|
|
- **Default provider:** **`custom:omni-101`** → `http://100.78.63.20:20128` (OmniRoute node `omniroute-1`, CT101/m93p) — verified live 2026-10-26, **re-verified 2026-11-07** (live `config.yaml` `model.default.provider: custom:omni-101`; a keyed `deepseek-paid-smart` chat probe against `100.78.63.20:20128` returns **HTTP 200**). The `custom:omni` provider key does not exist in `custom_providers`. ⚠ **Corrected 2026-11-07:** the old claim that *"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` shows **no job pinned to the bare `custom:omni`**; **all 21 provider-pinned jobs use `custom:omni-101`** (see [[systems/scheduled-tasks]])
|
|
- **Default model:** `smart-route` (previously `freellmapi-auto`)
|
|
- **Fallback providers** (live chain, re-verified 2026-10-26): `nous` / `meituan/longcat-2.5-preview:free` → `opencode-zen` / `deepseek-v4.1-flash` → `custom:guacano` / `deepseek-v4.1-flash` → `custom:litellm` / `kilo-reflection-beam-free` → `kilocode` / `poolside/laguna-m.1:free` → `custom` (`http://litellm:4000/v1`) / `or-inclusionai_ling-2_6_1t`. ⚠ The **first** entry is the dead `nous` credential — a fallover tries it first
|
|
- **Auxiliary models:** all eight `auxiliary:` slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) are **`no-think/freeapi/auto` on `custom:omni-101`** with a `freeapi/deepseek-v4-flash` → `kg/kilo-auto/free` fallback chain (moved 2026-10-25 19:13)
|
|
- See [[systems/model-providers]] for full breakdown.
|
|
|
|
## Memory
|
|
|
|
- **Backend:** Mnemosyne (local native)
|
|
- **Vector type:** int8
|
|
- **Shared surface:** `data/shared/mnemosyne.db`
|
|
- See [[systems/memory-backend]] for details.
|
|
|
|
## Scheduled Tasks
|
|
|
|
- **Cron jobs:** **28 — 25 enabled / 3 disabled** (verified live `jobs.json` 2026-10-26). **5 non-clean `last_status`**: `0e753e9720fd` (`error`), `fad5a86979d4` + `066bb5191280` (`delivery_failed`), `08eb86f02d37` (`delivery_queued`, empty `last_error`), `c0a9fa314542` (`error`, streak 1)
|
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
|
- Notable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup
|
|
- See [[systems/scheduled-tasks]] for full table.
|
|
|
|
## Messaging
|
|
|
|
- Telegram, Discord, Slack, Email, Mattermost, ntfy
|
|
- **Mattermost is currently unreachable** — the gateway logs `mattermost connect timed out after 30s, next retry in 300s` (re-probed live 2026-10-26) and the Mattermost host does not answer: CT255 is `stopped` on `pve` and its Tailscale peer has been offline **36d**. Treat Mattermost delivery as down until probed.
|
|
- See [[systems/messaging-integrations]] for details.
|
|
|
|
## Key Differences from Chaos
|
|
|
|
| Aspect | Production (CT460) | Chaos (CT450) |
|
|
|--------|-------------------|---------------|
|
|
| Purpose | Stable daily operations | Experimental/testing |
|
|
| Cron jobs | **29 (26 enabled / 3 disabled)** (re-verified 2026-11-07) | 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460 (re-verified 2026-11-07) |
|
|
| Config changes | Requires verification | Safe to experiment |
|
|
| Memory | Production Mnemosyne | Separate state |
|
|
|
|
## Related
|
|
|
|
- [[agents/hermes-chaos]] — The experimental twin
|
|
- [[agents/nanobot]] — Scout agent on CT333
|
|
- [[systems/hermes-agent]] — Hermes Agent system details
|
|
- [[systems/current-profile]] — Active profile details
|
|
- [[infrastructure/hosts]] — Host inventory
|