snapshot: preserve central wiki state 2026-08-15 (7 modified + 1 new file)
This commit is contained in:
Tony0410
@@ -0,0 +1,52 @@
|
|||||||
|
# SCHEMA — Anthony's Systems Wiki
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Help Hermes understand and maintain Anthony's agents, services,
|
||||||
|
automations, model routing and technical decisions.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. Do not store passwords, API keys, recovery codes, or private SSH keys.
|
||||||
|
A credentials-map.md records where secrets live; never their values.
|
||||||
|
2. Preserve raw sources unchanged in raw/.
|
||||||
|
3. Update existing pages instead of creating duplicates.
|
||||||
|
4. Every page uses YAML frontmatter and links to at least two related pages.
|
||||||
|
5. Separate facts from procedures:
|
||||||
|
system page = what/where/configured;
|
||||||
|
runbook = how to recover;
|
||||||
|
concept = why;
|
||||||
|
decision = what we chose and why;
|
||||||
|
raw doc snapshots = official source.
|
||||||
|
6. Record important choices in decisions/ with revisit triggers.
|
||||||
|
7. Capture repeatable fixes in runbooks/, not articles.
|
||||||
|
8. Record tests and experiments in experiments/ with explicit verdicts.
|
||||||
|
9. Mark outdated material as deprecated; never silently delete.
|
||||||
|
10. Include dates and version markers on technical instructions.
|
||||||
|
11. Ask before writing more than ten pages at once.
|
||||||
|
|
||||||
|
## Frontmatter
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
title: Page Title
|
||||||
|
type: system | infrastructure | concept | decision | comparison | runbook | experiment | query
|
||||||
|
status: active | experimental | deprecated | unresolved | archived
|
||||||
|
created: YYYY-MM-DD
|
||||||
|
updated: YYYY-MM-DD
|
||||||
|
review_after: YYYY-MM-DD
|
||||||
|
confidence: high | medium | low
|
||||||
|
tags: [from taxonomy]
|
||||||
|
sources: [raw/doc]
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tag Taxonomy
|
||||||
|
|
||||||
|
- systems: hermes, nanobot, router, memory, browser, automation, infrastructure
|
||||||
|
- content: concept, comparison, decision, runbook, experiment, source
|
||||||
|
- status: active, experimental, deprecated, unresolved, archived
|
||||||
|
|
||||||
|
## Freshness
|
||||||
|
|
||||||
|
Pages older than 90 days without update should be reviewed.
|
||||||
|
`version_applies_to:` and `verified_on:` should appear on operational pages.
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
---
|
||||||
|
title: Camoufox vs Chrome
|
||||||
|
type: comparison
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [comparison, browser]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Camoufox vs Chrome
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Compare the two browser-automation options for Hermes's browser tool.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
| | Camoufox | Chrome |
|
||||||
|
|---|---|---|
|
||||||
|
| In use on this install | **Yes** — configured `cloud_provider` | No evidence found |
|
||||||
|
| Config | `browser.camofox.managed_persistence: true` (note: key is spelled `camofox` in config, not `camoufox`) | n/a |
|
||||||
|
| Driver | `cua-driver` v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | n/a |
|
||||||
|
| Display | Headed, via Xvfb `:99` + fluxbox — see [[systems/headless-server]] | n/a |
|
||||||
|
| `use_gateway` | `false` | n/a |
|
||||||
|
|
||||||
|
## Detail
|
||||||
|
|
||||||
|
### Camoufox (in use)
|
||||||
|
- **Verified:** `browser.cloud_provider: camofox`, `inactivity_timeout: 120s`, `allow_private_urls: true`.
|
||||||
|
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
|
||||||
|
- `cua-driver` bridges Hermes's computer-use tool calls to the browser.
|
||||||
|
|
||||||
|
### Chrome (not found in use)
|
||||||
|
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
|
||||||
|
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
This install uses **Camoufox exclusively**. There's no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[concepts/browser-automation]]
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
---
|
||||||
|
title: Hermes vs Nanobot
|
||||||
|
type: comparison
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [comparison, hermes, nanobot]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes vs Nanobot
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Compare Hermes (this agent) with Nanobot, another agent process in the homelab, and correct a stale claim in [[systems/nanobot]].
|
||||||
|
|
||||||
|
## Correction to existing wiki content
|
||||||
|
`systems/nanobot.md` (written earlier the same day) states "No nanobot process running on this host" and "Unknown: whether nanobot should be redeployed on this host." **Both are superseded by direct verification during this pass:**
|
||||||
|
- Nanobot **is running**, just not on this host (ct460). It's on **LXC 333 ("clawtest")**, Tailscale IP `100.122.189.81`, process `nanobot` listening on port `8766`.
|
||||||
|
- ct333 also runs a `whatsapp-bridge.service` ("WhatsApp Bridge for Nanobot") — a systemd service, active.
|
||||||
|
- It's publicly reachable via the same Cloudflare Tunnel that fronts Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, and `nano-ssh.martinwa.org` → SSH on the same host.
|
||||||
|
- The earlier note "Nanobox on the PVE host at `192.168.178.39`" conflated Nanobot with `pve`'s own LAN IP (that address is Proxmox's web UI, per the tunnel's `pve.martinwa.org` entry) — an error, not a second Nanobot instance.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
||||||
|
|---|---|---|
|
||||||
|
| Status | Active, primary agent | Active, running (was previously assumed dormant) |
|
||||||
|
| Host | Dedicated LXC, `agent;ai;automation` tags | LXC tagged/named "clawtest" |
|
||||||
|
| Public exposure | `hermes.martinwa.org`, `hermes-desktop.martinwa.org` | `nanobot.martinwa.org`, `nano-ssh.martinwa.org` |
|
||||||
|
| Extra services | Stratos command centre, browser/VNC stack | WhatsApp bridge |
|
||||||
|
| Role today | Primary personal agent (this wiki's subject) | **Unclear** — running, but purpose/overlap with Hermes not documented anywhere found |
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
Both agents are live simultaneously, which the wiki didn't previously reflect. Worth a real decision: is Nanobot still needed alongside Hermes, or is it a leftover that should be retired (freeing ct333) or given an explicit distinct role? This comparison surfaces the question rather than answering it — no evidence either way was found on either host.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/nanobot]] — needs its "Current state" section updated to reflect this finding
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
title: Memory Backends
|
||||||
|
type: comparison
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-12
|
||||||
|
verified_on: 2026-08-12
|
||||||
|
confidence: high
|
||||||
|
tags: [comparison, memory]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Memory Backends
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Document Hermes's memory backend on this box, and record why the taxonomy this page assumed (Hindsight, Mnemosyne, file-backed) doesn't actually apply here.
|
||||||
|
|
||||||
|
## Correction (2026-07-22, per Anthony)
|
||||||
|
This wiki previously stated Hindsight was this box's predecessor memory backend, based on archived `hindsight-*` script names found under `~/.hermes/scripts/` and a "Hindsight clean memory bank" mention in `current-state.md`. **That was wrong — Hindsight has nothing to do with this Hermes box.** It was confused with a different Hermes instance (most likely ct450 "hermesagent", the older/original Hermes install still running in this homelab — see [[infrastructure/domains-and-tunnels]]). All Hindsight-specific pages (`systems/hindsight.md`, `decisions/use-hindsight-for-memory.md`, `runbooks/check-hindsight.md`) have been deleted rather than left as deprecated stubs, since they describe something that was never true of this box.
|
||||||
|
|
||||||
|
The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migration_watchdog.py.archived`, `honcho_to_hindsight_import.py.archived`) that had been sitting under `~/.hermes/scripts/` were removed on 2026-07-22 at Anthony's request, closing out the mix-up entirely.
|
||||||
|
|
||||||
|
## What's actually true here
|
||||||
|
- **Only backend ever verified on this box: Mnemosyne** — local native plugin (`~/.hermes/plugins/mnemosyne`), not a docker service. See [[systems/mnemosyne]] and [[systems/memory-backend]] for full config.
|
||||||
|
- No evidence of a file-backed-only memory mode being used either — it's Mnemosyne or nothing on this install.
|
||||||
|
|
||||||
|
## OpenViking evaluation (2026-08-12) — NOT adopted
|
||||||
|
- Anthony explored **OpenViking** (Volcengine/ByteDance context database, filesystem-style knowledge hierarchy, tiered retrieval, 6-category auto extraction) as a potential memory provider.
|
||||||
|
- **Not adopted:** `memory.provider` is single-slot — setting `hermes config set memory.provider openviking` would fully **replace** Mnemosyne, with no parallel/dual-active mode.
|
||||||
|
- Risk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot.
|
||||||
|
- **Verdict:** stay on Mnemosyne. Config verified `memory.provider: mnemosyne` (2026-08-12). No migration was performed.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[systems/memory-backend]]
|
||||||
|
- [[concepts/agent-memory]]
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
title: Model Routers
|
||||||
|
type: comparison
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-12
|
||||||
|
verified_on: 2026-08-12
|
||||||
|
confidence: high
|
||||||
|
tags: [comparison, routing]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Model Routers
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
| | LiteLLM | OmniRoute | OpenRouter |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Role in Hermes | **Default/primary** provider | **Fallback/auxiliary** provider | Not configured as a standalone provider |
|
||||||
|
| Endpoint | `http://litellm:4000/v1` | `http://omniroute:20128/v1` (public: `omniroute.martinwa.org`) | n/a |
|
||||||
|
| Advertised model count | ~1512 | ~3430 | n/a |
|
||||||
|
| Routing style | Straight proxy | Combo strategy, LKGP scoring, session affinity | n/a |
|
||||||
|
| Aux roles used for | — | `skills_hub`, `approval`, `mcp` | — |
|
||||||
|
| Config presence | `custom_providers` in `~/.hermes/config.yaml` | same | absent from `custom_providers`; only referenced via a kept [[decisions/keep-openrouter-as-fallback]] decision |
|
||||||
|
|
||||||
|
## Detail
|
||||||
|
|
||||||
|
### LiteLLM — primary
|
||||||
|
Default provider for main chat traffic (`freellmapi-auto` default model resolves through it). See [[systems/model-providers]].
|
||||||
|
|
||||||
|
### FreeLLMAPI — the router behind `freellmapi-auto`
|
||||||
|
Unified LLM router at `100.82.40.23:3001` (also `freellmapi.martinwa.org`). Verified 2026-08-12:
|
||||||
|
- **3-axis weighted scoring:** Reliability + Speed + Intelligence, with user-configurable weights.
|
||||||
|
- **Dynamic guardrail multiplier:** `Final score = weighted avg × (free-quota headroom × live rate-limit penalty)`. Below 1.0 means the model is held back.
|
||||||
|
- **Penalty Inspector ("Router Pressure"):** granular penalty factors (e.g. `score × 0.5`) from recent 4xx/5xx, cooldowns, RPM/RPD limits, monthly token caps.
|
||||||
|
- **Capability-aware routing:** filters candidates on whether the request has images (vision) or tool calls.
|
||||||
|
- Exposes `/api/fallback/routing`, `/api/fallback/penalty-inspector`, `/api/fallback/token-usage`.
|
||||||
|
This is the router that resolves Hermes's **default model** `freellmapi-auto` — distinct from OmniRoute's LKGP (below).
|
||||||
|
|
||||||
|
### OmniRoute — fallback / auxiliary
|
||||||
|
Not used for Hermes's main chat path but handles specific auxiliary roles (`skills_hub`, `approval`, `mcp`) and serves as the fallback provider. Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what's actually resolvable — combo discovery vs. resolution can differ (documented known issue). See [[systems/omniroute]].
|
||||||
|
|
||||||
|
### OpenRouter — kept only as a fallback decision, not wired in
|
||||||
|
**Verified:** absent from `custom_providers` in the live config. A decision page ([[decisions/keep-openrouter-as-fallback]]) records intent to retain it as a fallback option, but there's no live routing through it today — see [[systems/openrouter]] for the "previously used, largely replaced" history.
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
Two routers are actually live (LiteLLM primary, OmniRoute fallback/auxiliary); OpenRouter is a retained *intent*, not a working path, as of 2026-07-22. If [[decisions/keep-openrouter-as-fallback]] is still desired, it needs an actual `custom_providers` entry — currently it's fallback-in-name-only.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/model-providers]]
|
||||||
|
- [[concepts/model-routing]]
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
title: Steel Browser vs Camoufox
|
||||||
|
type: comparison
|
||||||
|
status: active
|
||||||
|
created: 2026-07-29
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [comparison, browser, steel, camofox, decision]
|
||||||
|
sources: [live-probe, session-history]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Steel Browser vs Camoufox
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Compare the two browser-automation backends available to Hermes, including the Jul 29–30 experiment and the decision to revert.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
| | Steel Browser (CT201) | Camoufox (CT450) |
|
||||||
|
|---|---|---|
|
||||||
|
| **In use** | Disabled since 2026-07-30 | **Yes** — primary backend |
|
||||||
|
| **Engine** | Chromium 140 (Playwright) | Camoufox (Firefox-based) |
|
||||||
|
| **Connection** | CDP (`ws://` via port 9223) | REST API (localhost:9093) |
|
||||||
|
| **API** | REST at `:3000` — create/destroy/query sessions | Camoufox REST + CDP WebSocket |
|
||||||
|
| **Headless** | Native headless | Xvfb + VNC (headed, anti-fingerprint) |
|
||||||
|
| **Container** | Docker on CT201 | Systemd service on CT450 |
|
||||||
|
| **Stealth** | Standard Chromium | Anti-detection fingerprinting |
|
||||||
|
| **Session viewer** | Built-in UI at `/steel` | VNC at `/vnc-camofox` |
|
||||||
|
| **Plugin** | `browser-steel` plugin (disabled) | Native Hermes `browser.cloud_provider: camofox` |
|
||||||
|
|
||||||
|
## The Steel Experiment (Jul 29–30)
|
||||||
|
|
||||||
|
### Why it was tried
|
||||||
|
- Steel promised full CDP access for richer browser control
|
||||||
|
- Built-in session viewer and management UI
|
||||||
|
- Docker-based, clean separation from Hermes host
|
||||||
|
|
||||||
|
### Timeline
|
||||||
|
1. **Jul 29:** Steel deployed on CT201 via Docker Compose, Chromium 140. Hermes configured with `browser.cloud_provider: steel`, `browser-steel` plugin enabled.
|
||||||
|
2. **Jul 29 (later):** CDP connectivity issues surfaced — Hermes was reaching `0.0.0.0` from the plugin's internal CDP URL. Required `_normalize_cdp_url()` patch in the Steel plugin to remap to `100.96.244.39`.
|
||||||
|
3. **Jul 30:** DeepSeek V4 Flash 0731 went live and rejected tool names that didn't match `^[a-zA-Z0-9_-]+$`. The Steel plugin's tools (`steel_scrape`, `steel_session_options`) were fine, but the route through OmniRoute → Steel → CDP was unreliable with empty-stream errors.
|
||||||
|
4. **Jul 30 (evening):** Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, `CAMOFOX_URL` and `CAMOFOX_API_KEY` uncommented in `.env`. Steel left running on CT201 but Hermes no longer routes through it.
|
||||||
|
|
||||||
|
### What broke
|
||||||
|
- CDP URL normalization required a plugin patch to handle Docker's internal networking
|
||||||
|
- Empty-stream errors from `opencode-go/deepseek-v4-flash` provider path — the combo strategy does NOT iterate on empty responses, they pass to Hermes's own fallback chain (the #1 cause of "combo works but Hermes falls back")
|
||||||
|
- Tool name validation on DeepSeek 0731 was stricter — this affected all providers, not just Steel
|
||||||
|
|
||||||
|
### Lesson learned
|
||||||
|
Steel is a capable backend but adds a network hop and another plugin layer. Camoufox runs locally on CT450 with fewer failure modes. The `browser-steel` plugin still exists (disabled) if needed in future.
|
||||||
|
|
||||||
|
## Detail
|
||||||
|
|
||||||
|
### Camoufox (current primary)
|
||||||
|
- **Deployed:** Persistent systemd service on CT450
|
||||||
|
- **Connection:** REST API at `localhost:9093`, profile `anthony`, sessionKey `anthony`
|
||||||
|
- **Key advantage:** Runs on the same host as Hermes — no network hop, no container networking to debug
|
||||||
|
- **Auth:** Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed
|
||||||
|
- **Limitation:** Headed via Xvfb, consumes display resources
|
||||||
|
|
||||||
|
### Steel Browser (disabled but available)
|
||||||
|
- **Deployed:** 2026-07-29 on CT201, Docker, custom `steel-browser-api:fixed` image
|
||||||
|
- **Status:** Still running on CT201, Hermes not connected
|
||||||
|
- **CDP endpoint:** `http://100.96.244.39:9223`
|
||||||
|
- **REST API:** `http://100.96.244.39:3000`
|
||||||
|
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
|
||||||
|
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled, `plugin_steel` toolset flagged `requires_health=steel`
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[concepts/browser-automation]]
|
||||||
|
- [[decisions/rollback-steel-to-camofox]]
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
---
|
||||||
|
title: Agent Memory
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [concept, memory]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Agent Memory
|
||||||
|
|
||||||
|
## Idea
|
||||||
|
How an agent retains facts/context beyond a single conversation turn or session — what gets stored, when it's flushed, and how it's retrieved later.
|
||||||
|
|
||||||
|
## How this install implements it
|
||||||
|
- **Backend:** Mnemosyne, local native (not a separate docker service) — see [[systems/mnemosyne]], [[comparisons/memory-backends]].
|
||||||
|
- **Verified settings:** memory enabled `true`; message char limit `5000`; user-profile char limit `3000`; flush every `6` turns; `auto_sleep: true`; default scope `session`; vector type `int8`; shared surface `data/shared/mnemosyne.db`.
|
||||||
|
- **Correction (2026-07-22, per Anthony):** this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See [[comparisons/memory-backends]] for the full correction.
|
||||||
|
|
||||||
|
## Model
|
||||||
|
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
||||||
|
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/memory-backend]]
|
||||||
|
- [[comparisons/memory-backends]]
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
---
|
||||||
|
title: Browser Automation
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [concept, browser, camofox, steel]
|
||||||
|
sources: [live-probe, session-history]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Browser Automation
|
||||||
|
|
||||||
|
## Idea
|
||||||
|
Giving an agent a real, controllable web browser — as opposed to just an HTTP-fetch tool — so it can handle JS-heavy pages, logins, and visual verification.
|
||||||
|
|
||||||
|
## How this install implements it
|
||||||
|
- **Engine:** Camoufox (Firefox-based, fingerprint-resistant) — see [[comparisons/camoufox-vs-chrome]] for why this over stock Chrome.
|
||||||
|
- **Connection:** REST API at `localhost:9093` (profile `anthony`) + CDP WebSocket at `localhost:9377`
|
||||||
|
- **Display:** headed, not truly headless — runs under `Xvfb :99` (1600×1200×24) with `fluxbox` as window manager, viewable live over VNC (see [[systems/headless-server]]).
|
||||||
|
- **Config:** `cloud_provider: camofox`, `camera_provider: camofox` in `~/.hermes/config.yaml`
|
||||||
|
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc. No cookie files needed.
|
||||||
|
|
||||||
|
## Why headed-under-Xvfb instead of headless
|
||||||
|
Running a real X display (rather than a headless browser flag) lets a human watch/intervene via VNC and tends to reduce automation-detection fingerprints compared to headless-mode flags — consistent with using Camoufox in the first place. The tradeoff, flagged in [[systems/headless-server]], is that the VNC exposing this display currently has no password.
|
||||||
|
|
||||||
|
## The Steel Experiment (Jul 29–30)
|
||||||
|
Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offered full CDP access and a built-in session viewer, but was reverted after:
|
||||||
|
- CDP URL normalization issues (Docker internal networking required a plugin patch)
|
||||||
|
- Empty-stream errors from the `opencode-go/deepseek-v4-flash` provider path
|
||||||
|
- DeepSeek 0731's stricter tool name validation affected all providers
|
||||||
|
|
||||||
|
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin is still present but disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
|
- [[comparisons/camoufox-vs-chrome]]
|
||||||
|
- [[systems/headless-server]]
|
||||||
|
- [[concepts/tool-calling]]
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
---
|
||||||
|
title: Model Routing
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [routing, model, inference]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Model Routing
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
How Hermes selects providers, applies combos, and enforces context limits.
|
||||||
|
|
||||||
|
## Current Flow
|
||||||
|
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
||||||
|
- Default model: `freellmapi-auto`
|
||||||
|
- OmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression
|
||||||
|
|
||||||
|
## Context Enforcement
|
||||||
|
- Three layers: model native → combo `data.config.context_length` → Hermes client cap
|
||||||
|
- Effective context is the minimum of the three
|
||||||
|
|
||||||
|
## Combo Behavior
|
||||||
|
- `priority` retries same provider up to `maxRetries`, then errors; it does NOT fall through
|
||||||
|
- `fill-first` walks the list on failure
|
||||||
|
- `auto`/`lkgp` recalculates per request using health scoring
|
||||||
|
- Same-provider entries in a combo share rate-limit buckets
|
||||||
|
- Session affinity caches provider in `session_model_history`
|
||||||
|
|
||||||
|
## Aliases and Mappings
|
||||||
|
- Model aliases resolved via `key_value` table (`modelAliases` namespace)
|
||||||
|
- Wildcard mappings via `model_combo_mappings`
|
||||||
|
- OpenCode/Zen entries may resolve back to `opencode-zen` provider internally
|
||||||
|
|
||||||
|
## History
|
||||||
|
- User-provided: OpenRouter was used previously; provider lists now centered on LiteLLM + OmniRoute
|
||||||
|
- User-provided: `persist_switch_by_default=false`; `/model` is per-session unless `--global` used
|
||||||
|
- Inferred: Configuration undergoes periodic corrections related to vision, model fallback, and browser backend tuning
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[model-providers]]
|
||||||
|
- [[systems/omniroute]]
|
||||||
|
- [[openrouter]]
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
---
|
||||||
|
title: Persistent Shells
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [concept, terminal]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Persistent Shells
|
||||||
|
|
||||||
|
## Idea
|
||||||
|
A background shell session that survives across separate Hermes turns/requests, instead of spawning and tearing down a fresh shell each time — so state (working directory, exported variables, long-running processes) carries over.
|
||||||
|
|
||||||
|
## How this install implements it
|
||||||
|
- **Verified (`~/.hermes/config.yaml`):** `terminal.persistent_shell: true`
|
||||||
|
- **Backend:** local (not containerized separately from the agent process)
|
||||||
|
- **Working directory:** `/home/hermes/workspace`
|
||||||
|
- **Timeout:** 180s
|
||||||
|
- **Home mode:** auto
|
||||||
|
- **Container CPU:** 1
|
||||||
|
|
||||||
|
## Why it matters here
|
||||||
|
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/terminal-backend]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[concepts/tool-calling]]
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
---
|
||||||
|
title: Tool Calling
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [concept, tool-calling]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Tool Calling
|
||||||
|
|
||||||
|
## Idea
|
||||||
|
How Hermes decides to invoke a tool, and — more specifically for this page — how it detects and stops itself when tool use goes wrong (loops, repeated failures).
|
||||||
|
|
||||||
|
## How this install implements guardrails
|
||||||
|
**Verified (`~/.hermes/config.yaml`, `tool_loop_guardrails:`):**
|
||||||
|
```
|
||||||
|
warnings_enabled: true
|
||||||
|
hard_stop_enabled: false
|
||||||
|
warn_after:
|
||||||
|
exact_failure: 2
|
||||||
|
same_tool_failure: 3
|
||||||
|
idempotent_no_progress: 2
|
||||||
|
hard_stop_after:
|
||||||
|
exact_failure: 5
|
||||||
|
same_tool_failure: 8
|
||||||
|
idempotent_no_progress: 5
|
||||||
|
```
|
||||||
|
- Warnings trigger well before any hard stop (2–3 repeats vs 5–8).
|
||||||
|
- **`hard_stop_enabled: false`** — Hermes will *warn* about a stuck tool loop but will not currently force-stop itself. That's a deliberate-looking but consequential setting: a genuinely stuck loop only gets a warning, not an automatic halt.
|
||||||
|
|
||||||
|
## Why it matters here
|
||||||
|
This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runbook's "agent loops on one tool" symptom is exactly what `warn_after`/`hard_stop_after` are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[concepts/persistent-shells]]
|
||||||
|
- [[runbooks/troubleshoot-tool-loop]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
---
|
||||||
|
title: Current State
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-14
|
||||||
|
review_after: 2026-10-22
|
||||||
|
confidence: high
|
||||||
|
tags: [system]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Current State
|
||||||
|
Updated: 2026-08-14
|
||||||
|
|
||||||
|
## Attested facts
|
||||||
|
- **Config version:** 33 — Verified, `~/.hermes/config.yaml`, 2026-07-25
|
||||||
|
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
|
||||||
|
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
|
||||||
|
- **Main provider:** `custom:litellm` → `http://litellm:4000/v1` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
||||||
|
- **Default model:** `freellmapi-auto` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
||||||
|
- **Fallback/aux provider:** `custom:omni` → `http://omni:20128/v1` — Verified, config, 2026-08-13 (migrated from `custom:omniroute`, CT600)
|
||||||
|
- **OmniRoute host:** `omni.kangaroo-eel.ts.net` → `100.93.204.4` — Verified, DNS resolution, 2026-08-13
|
||||||
|
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
|
||||||
|
- **Memory enabled:** true — Verified, config, 2026-07-22
|
||||||
|
- **Streaming enabled:** true — Verified, config, 2026-07-22
|
||||||
|
- **Dashboard public URL:** `https://hermes.martinwa.org/` — Verified, config, 2026-07-22
|
||||||
|
- **Pet config:** enabled/slug null — Verified, config, 2026-07-22
|
||||||
|
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-02)
|
||||||
|
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||||||
|
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
|
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
|
||||||
|
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
|
||||||
|
- **Uber Eats report upgraded:** Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with `camofox-browser-automation` + `browser-backend-verification` skills.
|
||||||
|
- **GitHub Trending Radar — Daily Digest removed:** Stopped by user on 2026-08-02. Weekly roundup still active.
|
||||||
|
- **Ops profile renamed:** Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
|
||||||
|
- **Wiki audit cron job created:** Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
|
||||||
|
- **Ops profile Mnemosyne enabled:** Mnemosyne plugin enabled on ops profile with `profile_isolation: true` — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: `~/.hermes/profiles/ops/config.yaml` (`memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`).
|
||||||
|
- **OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04):** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
|
||||||
|
- **OmniRoute three-layer fix (2026-08-07):** npm package `.env` override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See [[systems/omniroute]].
|
||||||
|
- **Nanobot status degraded (2026-08-07):** CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See [[systems/nanobot]].
|
||||||
|
- **CT460 memory bumped (2026-08-07):** 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-08)
|
||||||
|
- **Cron job model repinning:** 3 jobs repinned from dead `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker repinned from dead `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named cron jobs (e.g. `vertex/gemini-2.5-flash`, `gpt-5.6-terra`, `deepseek-v4-flash-free`) repinned to `smart-route`. Daily Marketing Job Radar swapped to `gpt-5.6-terra`.
|
||||||
|
- **Hermes UI vite config revert:** Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build now serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
|
||||||
|
- **Phase One estate review completed:** Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, and CT333 Nanobot has a live `jobs.json` with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-09)
|
||||||
|
- **Portainer password reset (CT245):** Admin user is `anthony` (not `admin`). Password was forgotten; reset via `docker run --rm -v portainer_data:/data portainer/helper-reset-password` after stopping the Portainer container. Verified end-to-end with `POST /api/auth` → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on `:9443` (see [[systems/homepage-dashboard]]). Recovery procedure: stop container → run helper image → restart → verify with auth API.
|
||||||
|
- **SFTPGo security review (CT299):** Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path `//` resolves to `/` — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is `root`, (4) quota value `F1` on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-10)
|
||||||
|
- **Cron job failures — session storage issue:** Four cron jobs failed on 2026-08-10 with two distinct error patterns:
|
||||||
|
- **TimeoutError (idle for 601–602s):** "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
|
||||||
|
- **RuntimeError (session storage):** "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
|
||||||
|
- **Root cause:** Disk at 83% (11GB free on 65GB). `state.db` is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
|
||||||
|
- **Impact:** News digests and GitHub Trending did not deliver. Job Radar did not run.
|
||||||
|
- **See:** [[systems/scheduled-tasks]] Known Issues section.
|
||||||
|
- **OpenCode WebSocket proxy errors:** Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, `opencode serve`), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-14)
|
||||||
|
- **OmniRoute factory reset on CT601:** Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See [[systems/omniroute]].
|
||||||
|
- **Hermes default model changed:** `auto/best-free` → `resilient-free` in `~/.hermes/config.yaml`. `auto/best-free` was misconfigured (no combo definition, routed to paid `gpt-5.6-sol`).
|
||||||
|
- **Multiple free combos broken by upstream model ID changes:** OpenRouter Poolside `laguna-s-2-1:free` → `laguna-s-2.1:free` (dash→dot); Groq `llama-4-scout-17b-16e-instruct` retired. Affected: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`.
|
||||||
|
- **Guanaco router updated:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See [[systems/guanaco]].
|
||||||
|
- **GPT-5.6 Luna research:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See [[systems/model-providers]].
|
||||||
|
- **Hermes UI restart:** Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
|
||||||
|
- **LiteLLM routing issue identified:** Hermes was previously pointed at LiteLLM (`http://litellm:4000/v1`, model `free-auto`) causing 6–26s per call. OmniRoute `auto/best-free` tested at 265ms. Config now points at OmniRoute.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-13)
|
||||||
|
- **OmniRoute migrated CT600 → CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Backup: `~/.hermes/config.yaml.bak.omniroute-removal`. See [[systems/omniroute]].
|
||||||
|
- **Evening stand-down briefing weather bugfix:** Weather collector returns a plain string (`+16°C Sunny`), but the three dashboard widgets (`evening.html`, `evening-grid.html`, `evening-briefing.html`) read object fields → `undefined`. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[systems/scheduled-tasks]]
|
||||||
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
|
- [[decisions/rollback-steel-to-camofox]]
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
---
|
||||||
|
title: Decisions
|
||||||
|
type: collection
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [decisions]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Decisions
|
||||||
|
|
||||||
|
Stable strategic and config decisions, with evidence and status.
|
||||||
|
|
||||||
|
## Current decisions
|
||||||
|
|
||||||
|
- [[decisions/keep-openrouter-as-fallback]]
|
||||||
|
- [[decisions/persistent-shell-enabled]]
|
||||||
|
- [[decisions/rollback-steel-to-camofox]]
|
||||||
|
|
||||||
|
## History
|
||||||
|
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22
|
||||||
|
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
|
||||||
|
- **Unknown:** Exact date each listed decision was made; exact alternatives considered
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[current-state]]
|
||||||
|
- [[model-routing]]
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
---
|
||||||
|
title: Keep OpenRouter as fallback
|
||||||
|
type: decision
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
review_after: 2026-08-22 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
||||||
|
confidence: high
|
||||||
|
tags: [routing, provider, decision]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
# Keep OpenRouter as fallback
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
OpenRouter must remain available as a fallback provider, even if a self-hosted router becomes primary.
|
||||||
|
|
||||||
|
## Reasons
|
||||||
|
- Free models are important.
|
||||||
|
- Broad model coverage.
|
||||||
|
- Previous migrations disrupted provider availability.
|
||||||
|
- A fallback reduces the chance of Hermes becoming unusable.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
- Direct providers only
|
||||||
|
- OmniRoute exclusively
|
||||||
|
- Local models exclusively
|
||||||
|
|
||||||
|
## Revisit when
|
||||||
|
A self-hosted router has operated reliably for at least 30 days.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[model-routing]]
|
||||||
|
- [[systems/omniroute]]
|
||||||
|
- [[systems/openrouter]]
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
---
|
||||||
|
title: Persistent shell enabled
|
||||||
|
type: decision
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
review_after: 0001-01-01
|
||||||
|
confidence: high
|
||||||
|
tags: [automation, shell, decision]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
# Persistent shell enabled
|
||||||
|
## Decision
|
||||||
|
Enable persistent shells for Hermes.
|
||||||
|
|
||||||
|
## Reasons
|
||||||
|
- Preserve terminal state across turns
|
||||||
|
- Avoid re-activating virtualenvs and aliases every call
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
- Always transient shell
|
||||||
|
- Manual re-activation by user
|
||||||
|
|
||||||
|
## Revisit when
|
||||||
|
Use case becomes critical again.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[persistent-shells]]
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
title: Rollback Steel to Camoufox
|
||||||
|
type: decision
|
||||||
|
status: active
|
||||||
|
created: 2026-08-02
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [decision, browser, steel, camofox, rollback]
|
||||||
|
sources: [session-history, live-config]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Decision: Rollback Steel Browser → Camoufox
|
||||||
|
|
||||||
|
## Date
|
||||||
|
2026-07-30
|
||||||
|
|
||||||
|
## Context
|
||||||
|
Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automation backend, replacing Camoufox. It ran for approximately 24 hours before being reverted.
|
||||||
|
|
||||||
|
## Options Considered
|
||||||
|
| Option | Pros | Cons |
|
||||||
|
|---|---|---|
|
||||||
|
| **Keep Steel** | Full CDP access, session viewer UI, Docker isolation | CDP URL normalization issues, network hop, empty-stream errors |
|
||||||
|
| **Revert to Camoufox** | Local (no network hop), already authenticated, fewer failure modes | Headed display consumes resources, no built-in session viewer |
|
||||||
|
| **Both (Steel primary, Camoufox fallback)** | Redundancy | Complexity, two backends to maintain, unclear which one Hermes uses per-request |
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
**Revert to Camoufox as primary.** Steel plugin disabled but left running on CT201.
|
||||||
|
|
||||||
|
## Rationale
|
||||||
|
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
|
||||||
|
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
|
||||||
|
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
|
||||||
|
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
- **Positive:** Browser automation is stable again. Uber Eats report ran successfully via Camofox on 2026-08-01.
|
||||||
|
- **Negative:** Steel's session viewer UI and full CDP access are no longer available to Hermes.
|
||||||
|
- **Neutral:** Steel still runs on CT201 — can be re-enabled by uncommenting `.env` lines and enabling the plugin.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
- Camoufox verified working Jul 30: `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s
|
||||||
|
- Uber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51
|
||||||
|
- Steel plugin `browser-steel` still present at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[concepts/browser-automation]]
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
title: Example experiment
|
||||||
|
type: experiment
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
tags: [experiment]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
# Example experiment
|
||||||
|
|
||||||
|
## Question
|
||||||
|
|
||||||
|
## Baseline
|
||||||
|
|
||||||
|
## Configuration tested
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
Pending.
|
||||||
|
## Next action
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
---
|
||||||
|
title: Experiments
|
||||||
|
type: collection
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [experiments]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Experiments
|
||||||
|
|
||||||
|
Non-canonical tests, tuning passes, and change attempts.
|
||||||
|
|
||||||
|
## Known experiments
|
||||||
|
|
||||||
|
- `pre-vision-fix-20260718-195712` config snapshot — Verified, backup file exists, 2026-07-18
|
||||||
|
- `pre-model-fix` config snapshot — Verified, backup file exists
|
||||||
|
- `pre-camofox-tuning` config snapshot — Verified, backup file exists, 2026-07-20
|
||||||
|
|
||||||
|
## Outcomes
|
||||||
|
- **Verified:** Backups exist with the above names
|
||||||
|
- **Unknown:** Whether each experiment succeeded, failed, or was reverted
|
||||||
|
- **Excluded:** No secrets, tokens, or full config dumps are stored here
|
||||||
|
|
||||||
|
## History
|
||||||
|
- **Verified:** Three named pre-change backups present as of 2026-07-22
|
||||||
|
- **Inferred:** Backups were taken before optional config changes and left intact
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[runbooks/update-hermes-safely]]
|
||||||
|
- [[current-state]]
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# mnemosyne-tools-none — 2026-07-29
|
||||||
|
|
||||||
|
## FIXED: `memory.mnemosyne.tools: None` in config.yaml
|
||||||
|
|
||||||
|
**Root cause:** `~/.hermes/config.yaml` had `tools: None` under the mnemosyne section (line 183). The memory manager interprets `None` as a literal tool name filter and fails schema loading with:
|
||||||
|
```
|
||||||
|
WARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.
|
||||||
|
```
|
||||||
|
Fallback: the next line registers all 40 tools anyway — no data loss, just noise.
|
||||||
|
|
||||||
|
**Fix applied:** Removed the `tools: None` line. Config now defaults to all tools (correct behaviour).
|
||||||
|
|
||||||
|
**Timeline:** 86 occurrences logged between Jul 28–29, 19 unique sessions affected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# relay-scope-corruption — 2026-07-29
|
||||||
|
|
||||||
|
## SYMPTOM
|
||||||
|
```
|
||||||
|
WARNING agent.relay_runtime: Hermes Relay turn finalization failed
|
||||||
|
RuntimeError: invalid argument: scope handle is not at the top of the stack
|
||||||
|
File relay_runtime.py:644 end_turn → run_in_session → invoke
|
||||||
|
File nemo_relay/scope.py:144 pop()
|
||||||
|
```
|
||||||
|
|
||||||
|
## DETAILS
|
||||||
|
- Hermes Agent 0.19.0, nemo_relay 0.6.0
|
||||||
|
- 3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)
|
||||||
|
- Sessions: `20260723_024537_95c0e6f7`, `20260728_233758_0c6b3c0d`, `20260729_070054_b8452262`
|
||||||
|
- Exception is caught (WARNING, not ERROR). Turn finalization may be incomplete.
|
||||||
|
- No recurrence after 08:10.
|
||||||
|
|
||||||
|
## LIKELY CAUSE
|
||||||
|
Scope push/pop ordering mismatch. Something pushes a scope that isn't at the top when `end_turn` tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.
|
||||||
|
|
||||||
|
## REPORTED
|
||||||
|
Filed (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.
|
||||||
|
|
||||||
|
## MONITORING
|
||||||
|
Watch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.).
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
---
|
||||||
|
title: Mnemosyne Automatic Capture Failure
|
||||||
|
type: incident
|
||||||
|
status: active
|
||||||
|
created: 2026-08-01
|
||||||
|
updated: 2026-08-01
|
||||||
|
verified_on: 2026-08-01
|
||||||
|
confidence: high
|
||||||
|
tags: [memory, mnemosyne, hermes, incident]
|
||||||
|
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mnemosyne Automatic Capture Failure — 2026-08-01
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
Ordinary Hermes conversations were not being written to Mnemosyne `working_memory`. Explicit memory tools, SQLite integrity, vectors, FTS indexes, consolidation, backups, and cron monitoring remained operational. The newest automatic/explicit working row before repair was approximately fourteen hours old despite active Hermes sessions.
|
||||||
|
|
||||||
|
The always-injected legacy files were also effectively full: `MEMORY.md` was 5,055 bytes against a 5,000-character limit and `USER.md` was 2,958 bytes against 3,000. Legacy writes had begun failing.
|
||||||
|
|
||||||
|
## Root Cause
|
||||||
|
Structured values under `memory.mnemosyne` in `~/.hermes/config.yaml` had been serialized as strings instead of native YAML types. The critical value was:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sync_roles: "['user']"
|
||||||
|
```
|
||||||
|
|
||||||
|
The provider interpreted this as the literal invalid role `['user']`, so turn synchronization completed without persisting a user row. Other malformed values included `skip_contexts`, `reflect`, `sleep_threshold`, and `ignore_patterns`.
|
||||||
|
|
||||||
|
## Repair
|
||||||
|
- Backed up config to `/home/hermes/.hermes/backups/mnemosyne-capture-fix-20260801_180249/config.yaml.before`.
|
||||||
|
- Rewrote structured Mnemosyne settings with native YAML types.
|
||||||
|
- Set `profile_isolation: true` because the installation has multiple Hermes profiles.
|
||||||
|
- Preserved `sync_roles: [user]` and the intended skipped contexts.
|
||||||
|
- Ran `hermes config check` successfully.
|
||||||
|
- Created authoritative Mnemosyne canonical slots for user preferences and key workflows, correcting stale light-theme, numbered-Obsidian, and CT460 route entries.
|
||||||
|
- Backed up legacy files to `/home/hermes/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
||||||
|
- Reduced legacy injection from 8,013 bytes to 1,080 bytes while retaining essential always-on rules.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
A pre-fix CLI turn using marker `MNEMO_E2E_20260801_180113_27053` completed but produced no database row.
|
||||||
|
|
||||||
|
A post-fix CLI turn using marker `MNEMO_FIXED_20260801_180310_16583` wrote:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[USER] Reply with exactly OK. Memory capture test marker: MNEMO_FIXED_20260801_180310_16583
|
||||||
|
```
|
||||||
|
|
||||||
|
The row was verified directly in `~/.hermes/mnemosyne/data/mnemosyne.db`. Final diagnostics reported zero failed checks, complete working-vector coverage, zero orphan embeddings, 1,482 working rows, and 463 episodic vectors.
|
||||||
|
|
||||||
|
## Remaining Activation Step
|
||||||
|
The already-running Telegram gateway/provider instance was initialized before the config repair. Run `hermes gateway restart` from an external terminal to load the corrected typed configuration for messaging-platform turns. The gateway cannot safely restart itself from one of its own child tool calls.
|
||||||
|
|
||||||
|
## Monitoring Note
|
||||||
|
The health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest `working_memory` row rather than lowering the threshold blindly and creating idle-period false positives.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[systems/memory-backend]]
|
||||||
|
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# [2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)
|
||||||
|
|
||||||
|
## Status
|
||||||
|
Mitigated — combos reworked; upstream still degraded.
|
||||||
|
|
||||||
|
## Symptom
|
||||||
|
On **both** Hermes instances (ops/Tusk + the other instance), turns using `deepseek-flash-free` (and `smart-route` via `good-free`) did one tool call, then went **silently** — no text, no error. "continue" → one more tool call → silent stop again.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
The free-DeepSeek upstreams behind the combos started returning **valid-but-empty HTTP 200s right after a tool_calls turn** (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began **2026-08-01 ~09:14 UTC**, still active. OmniRoute log markers:
|
||||||
|
|
||||||
|
```
|
||||||
|
[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
|
||||||
|
[ProxyEgress] <provider>/<conn-hash> ... status=error
|
||||||
|
```
|
||||||
|
|
||||||
|
Per-model evidence (counts in `storage.sqlite` era log):
|
||||||
|
- `238d5978/deepseek-v4-flash` — 63× empty-after-tool (conn `d4b9fee0`, 189 proxy errors)
|
||||||
|
- `238d5978/deepseek-v4-flash-0731` — 70× empty-after-tool
|
||||||
|
- `eb5af0da/deepseek-v4-flash-thinking:free` (conn `f8169018`) — 504 proxy errors (combo #1 slot)
|
||||||
|
- `nvidia` deepseek + `opencode-zen` deepseek-free — noisy, errors
|
||||||
|
|
||||||
|
Strategy trap: both combos used `priority`, which retries the **same** broken connection and never fails over on an empty 200.
|
||||||
|
|
||||||
|
## Change applied (2026-08-04, ~03:30 AWST)
|
||||||
|
Edited `combos` in `/var/lib/omniroute/storage.sqlite` on CT600 (live DB, hot-reloads — no restart):
|
||||||
|
|
||||||
|
- **`deepseek-flash-free`** — strategy `priority` → `fill-first`; 8 → 7 models: healthy free first (`mimo-v2.5-free`, `laguna-s-2.1:free` ×2 providers, `step-3.7-flash`, `glm-5.2-cheap:free`), one deepseek (`deepseek-v4-flash-thinking:free`) demoted to last resort.
|
||||||
|
- **`good-free`** — strategy `priority` → `fill-first`; 15 → 12 models. **Dropped** the proven culprits: `238d5978/deepseek-v4-flash`, `nvidia/deepseek-ai/deepseek-v4-pro`, `nvidia/deepseek-ai/deepseek-v4-flash`. `opencode/deepseek-v4-flash-free` kept as last resort.
|
||||||
|
|
||||||
|
Backup of both pre-change rows: `/tmp/combo_backup_20260804.json` on the Hermes gateway LXC.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- Live API (`/v1/combos`) reflects both changes immediately (hot reload).
|
||||||
|
- 2-step tool-call round-trip through `deepseek-flash-free` returns proper final content.
|
||||||
|
- Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.
|
||||||
|
|
||||||
|
## Follow-ups
|
||||||
|
- Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
|
||||||
|
- Consider `auto`+explorationRate for self-tuning if free tier stays flaky.
|
||||||
|
- Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.
|
||||||
|
|
||||||
|
## Refs
|
||||||
|
- [[systems/omniroute]]
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# Anthony's Systems Wiki
|
||||||
|
> Operational knowledge for Anthony's agents, infrastructure,
|
||||||
|
> model routing and automation. Last updated: 2026-08-14
|
||||||
|
|
||||||
|
## Start here
|
||||||
|
- [[current-state]] — Current systems status
|
||||||
|
- [[systems/hermes-agent]] — Hermes Agent overview and version
|
||||||
|
- [[systems/current-profile]] — Active Hermes profile details
|
||||||
|
|
||||||
|
## Systems
|
||||||
|
### Core
|
||||||
|
- [[systems/hermes-agent]] — Agent version, install, config paths
|
||||||
|
- [[systems/current-profile]] — Active profile, default settings
|
||||||
|
- [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack)
|
||||||
|
- [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup
|
||||||
|
- [[systems/memory-backend]] — Mnemosyne vector memory, settings
|
||||||
|
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
||||||
|
- [[systems/omniroute]] — OmniRoute catalog, combos, and host path
|
||||||
|
- [[systems/openrouter]] — OpenRouter current state and history
|
||||||
|
- [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing
|
||||||
|
- [[systems/nanobot]] — Nanobot current state and history
|
||||||
|
- [[systems/browser-backend]] — Camoufox (primary) + Steel (disabled), CDP config
|
||||||
|
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
|
||||||
|
- [[systems/terminal-backend]] — Shell persistence, working directory
|
||||||
|
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||||
|
|
||||||
|
### Integrations
|
||||||
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||||
|
- [[systems/scheduled-tasks]] — 28 active cron jobs
|
||||||
|
- [[systems/skills-index]] — Installed skills inventory
|
||||||
|
|
||||||
|
### Monitoring & Dashboards
|
||||||
|
- [[systems/homepage-dashboard]] — Homepage dashboard (CT201)
|
||||||
|
- [[systems/home-assistant]] — Home Assistant integration
|
||||||
|
|
||||||
|
### Reports
|
||||||
|
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
|
||||||
|
- [[systems/obsidian]] — Obsidian vault structure and conventions
|
||||||
|
|
||||||
|
## Infrastructure
|
||||||
|
### Services
|
||||||
|
- [[infrastructure/hosts]] — Hosts, containers, important paths
|
||||||
|
- [[infrastructure/docker-services]] — Docker compose files
|
||||||
|
- [[infrastructure/credentials-map]] — Where credentials live (no values)
|
||||||
|
- [[infrastructure/network-map]] — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
|
||||||
|
- [[infrastructure/domains-and-tunnels]] — martinwa.org hostnames and Cloudflare Tunnel routing
|
||||||
|
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
|
||||||
|
|
||||||
|
### Networking
|
||||||
|
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers
|
||||||
|
- [[infrastructure/tailscale]] — Tailscale serve routes and VPN
|
||||||
|
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
||||||
|
|
||||||
|
## Concepts
|
||||||
|
- [[concepts/agent-memory]] — Memory model
|
||||||
|
- [[concepts/model-routing]] — Routing concepts
|
||||||
|
- [[concepts/persistent-shells]] — Terminal persistence
|
||||||
|
- [[concepts/browser-automation]] — Browser automation concepts
|
||||||
|
- [[concepts/tool-calling]] — Tool calling patterns
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
- [[decisions/index]] — Decision index
|
||||||
|
- [[decisions/keep-openrouter-as-fallback]] — Keep OpenRouter as fallback provider
|
||||||
|
- [[decisions/persistent-shell-enabled]] — Enable shell persistence
|
||||||
|
|
||||||
|
## Comparisons
|
||||||
|
- [[comparisons/camoufox-vs-chrome]] — Camoufox vs Chrome
|
||||||
|
- [[comparisons/hermes-vs-nanobot]] — Hermes vs Nanobot
|
||||||
|
- [[comparisons/memory-backends]] — Memory backend comparison
|
||||||
|
- [[comparisons/model-routers]] — Model router comparison
|
||||||
|
|
||||||
|
## Runbooks
|
||||||
|
- [[runbooks/multiplexer-setup]] — Multi-profile gateway multiplexer setup
|
||||||
|
- [[runbooks/diagnose-docker]] — Docker diagnosis
|
||||||
|
- [[runbooks/recover-docker-service]] — Docker service recovery
|
||||||
|
- [[runbooks/restart-hermes]] — Hermes restart procedure
|
||||||
|
- [[runbooks/restart-browser]] — Browser restart procedure
|
||||||
|
- [[runbooks/update-hermes-safely]] — Safe Hermes update
|
||||||
|
- [[runbooks/restore-openrouter]] — OpenRouter restoration
|
||||||
|
- [[runbooks/provider-health]] — Provider health check
|
||||||
|
- [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting
|
||||||
|
- [[runbooks/gateway-resource-resilience]] — Gateway resilience
|
||||||
|
- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure
|
||||||
|
|
||||||
|
## Raw
|
||||||
|
- [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
---
|
||||||
|
title: Backups
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [infrastructure, backups]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Backups
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
How Hermes's own configuration/state gets backed up, and what's missing.
|
||||||
|
|
||||||
|
## Current mechanism
|
||||||
|
- **`hermes backup` CLI** (verified via `hermes backup --help`): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).
|
||||||
|
- `--quick` / `-q`: fast snapshot of just critical state (`config`, `state.db`, `.env`, `auth`, `cron`)
|
||||||
|
- `--label` / `-l`: label for a quick snapshot
|
||||||
|
- `--output` / `-o`: custom output path (default `~/hermes-backup-<timestamp>.zip`)
|
||||||
|
- **Config keys** (`updates:` section of `~/.hermes/config.yaml`):
|
||||||
|
- `pre_update_backup: false` — Hermes does **not** automatically back up before self-updates
|
||||||
|
- `backup_keep: 5` — retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in [[runbooks/update-hermes-safely]])
|
||||||
|
- `non_interactive_local_changes: stash`
|
||||||
|
|
||||||
|
## Evidence on disk (2026-07-22)
|
||||||
|
- One manual full backup exists: `/home/hermes/hermes-backup-2026-07-22-180830.zip` (~235 MB), same day as wiki creation.
|
||||||
|
- `/home/hermes/wiki-personal-backups/` holds an earlier snapshot of the wiki itself (`SCHEMA.md`, `current-state.md`, `index.md`, `log.md` from 17:34, before later same-day wiki edits) — see [[runbooks/backup-wiki]].
|
||||||
|
- `/home/hermes/hermes-wiki/` exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.
|
||||||
|
- A dedicated memory-backup script exists: `/home/hermes/.hermes/scripts/mnemosyne-backup.sh` — see [[systems/mnemosyne]].
|
||||||
|
|
||||||
|
## Known gaps
|
||||||
|
- **No scheduled backup job found:** `crontab -l -u hermes` returns empty, and no systemd timer references `hermes` or `backup`. The one full backup on disk appears to have been triggered manually, not on a schedule.
|
||||||
|
- `pre_update_backup: false` means an in-place `hermes update` will not auto-snapshot first — the manual step in [[runbooks/update-hermes-safely]] step 1 is currently the only safety net.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/backup-wiki]]
|
||||||
|
- [[runbooks/update-hermes-safely]]
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
---
|
||||||
|
title: Credentials Map
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [credentials, security, infrastructure]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Credentials Map
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Document where credentials are stored and what service they belong to.
|
||||||
|
**No credential values are stored in this wiki.**
|
||||||
|
|
||||||
|
## Storage Locations
|
||||||
|
|
||||||
|
| Service | Credential Store | Location | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Telegram Bot | Hermes config | `~/.hermes/config.yaml` | Not exposed in wiki |
|
||||||
|
| Discord Bot | Hermes config | `~/.hermes/config.yaml` | Not exposed in wiki |
|
||||||
|
| Email | Hermes config | `~/.hermes/config.yaml` |-connected via IMAP/SMTP |
|
||||||
|
| Mattermost | Hermes config | `~/.hermes/config.yaml` | Connected |
|
||||||
|
| CF Access (mcp-portal) | Hermes config | `~/.hermes/config.yaml` | Client ID + Secret |
|
||||||
|
| LiteLLM | Hermes config | `~/.hermes/config.yaml` | API key |
|
||||||
|
| OmniRoute | Hermes config | `~/.hermes/config.yaml` | API key |
|
||||||
|
| Anthropic | Hermes config/OAuth | `~/.hermes/` | `.anthropic_oauth.json` |
|
||||||
|
| API Vault | Hermes vault | via `hermes-api-vault` skill | Notion-backed vault |
|
||||||
|
| Huawei Cloud | Terraform state / config | Unverified | |
|
||||||
|
| GitHub | Hermes config / gh CLI | `~/.config/gh/` or `~/.hermes/` | Unverified |
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- Credentials never written to this wiki
|
||||||
|
- API vault is first-stop for any key lookup
|
||||||
|
- If vault lacks it, surface the gap to user — do not fabricate
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[hosts]]
|
||||||
|
- [[model-providers]]
|
||||||
|
- [[messaging-integrations]]
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
title: Docker Services
|
||||||
|
type: infrastructure
|
||||||
|
status: mixed
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [docker, homelab, infrastructure]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Docker Services
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Containerized services supporting Hermes and the homelab.
|
||||||
|
|
||||||
|
## Status
|
||||||
|
- Docker runtime **not available** on this host (`docker ps` returned unavailable)
|
||||||
|
- Compose files confirmed present but execution state unverified
|
||||||
|
|
||||||
|
## Confirmed Compose Files
|
||||||
|
|
||||||
|
| File | Path | Services |
|
||||||
|
|---|---|---|
|
||||||
|
| Hermes Agent | `/home/hermes/.hermes/hermes-agent/docker-compose.yml` | hermes, hermes-dashboard |
|
||||||
|
| Hermes WebUI | `/home/hermes/hermes-webui/docker-compose.yml` | webui |
|
||||||
|
| ReadLater | `/home/hermes/workspace/readlater/docker-compose.yml` | readlater, edge-tts, kokoro-tts |
|
||||||
|
| Steel Browser | `/root/steel-browser/docker-compose.yml` (CT201) | steel-browser-api, steel-browser-ui |
|
||||||
|
|
||||||
|
## Service Definitions (from compose files)
|
||||||
|
|
||||||
|
### hermes-agent
|
||||||
|
- **Images:** `hermes-agent`
|
||||||
|
- **Containers:** `hermes`, `hermes-dashboard`
|
||||||
|
- **Volumes:** present (paths unverified)
|
||||||
|
- **Environment:** present (values unverified — secrets redacted)
|
||||||
|
|
||||||
|
### hermes-webui
|
||||||
|
- **Ports:** present (specific ports unverified)
|
||||||
|
- **Volumes:** present
|
||||||
|
|
||||||
|
### readlater
|
||||||
|
- **Container:** `readlater`
|
||||||
|
- **Ports:** present
|
||||||
|
- **Volumes:** present
|
||||||
|
- **Dependencies:** `edge-tts` (travisvn/openai-edge-tts), `kokoro-tts` (ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1)
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- Docker runtime unavailable on inspection host — service state unknown
|
||||||
|
- Specific ports, volumes, and env vars not extracted from compose files during this pass
|
||||||
|
|
||||||
|
## Additional Services (CT201 runtipi)
|
||||||
|
- **Homepage** — Dashboard at port 8082 (Tailscale: `https://runtipi.kangaroo-eel.ts.net:8443/`)
|
||||||
|
- **Manager** — Web UI for Homepage at port 8084
|
||||||
|
- See [systems/homepage-dashboard.md](../systems/homepage-dashboard.md) for full docs
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[docker-services]]
|
||||||
|
- [[infrastructure/hosts]]
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
---
|
||||||
|
title: Domains and Tunnels
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [infrastructure, network, cloudflare]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Domains and Tunnels
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Where `martinwa.org` subdomains point, and how they reach internal services.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
- **Verified:** All `*.martinwa.org` public hostnames are served by a single `cloudflared` daemon running on **ct207 (webserver)**, tunnel name `ct207-tunnel` (ID `339a6757-5574-4723-a16c-da4769535caa`), config at `/etc/cloudflared/config.yml` on ct207.
|
||||||
|
- **Verified:** `cloudflared` is not installed on this host (ct460/hermes) — Hermes has no local tunnel process to manage or restart. Any tunnel-side incident (hostname down, wrong target) must be fixed on ct207, not here.
|
||||||
|
- Two other Cloudflare tunnels exist in the same account (`Cloudo`, `Ifttt-ssh-webhook`, `livinity-liv`) but are unrelated to Hermes/ct207 and out of scope for this page.
|
||||||
|
|
||||||
|
## Hostnames relevant to Hermes
|
||||||
|
| Hostname | Target | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `hermes.martinwa.org` | `http://100.118.5.51:19119` | Public Hermes dashboard — nginx on ct460 proxies 19119→127.0.0.1:9119 |
|
||||||
|
| `hermes-desktop.martinwa.org` | `http://100.118.5.51:5174` | Hermes desktop companion; sets `Host: hermes.kangaroo-eel.ts.net` |
|
||||||
|
| `hermesagent.martinwa.org` | `http://100.98.205.112:9119` | The **older/original** Hermes agent (ct450 "hermesagent"), kept alive alongside the current ct460 instance |
|
||||||
|
| `ssh-hermes.martinwa.org` | `ssh://100.118.5.51:22` | SSH to this host over the tunnel |
|
||||||
|
| `mcp-portal.martinwa.org` | (not in ct207 ingress list; see note below) | Referenced in Hermes config as an MCP server URL (`https://mcp-portal.martinwa.org/mcp`) |
|
||||||
|
| `omniroute.martinwa.org` | `http://omniroute.kangaroo-eel.ts.net:20128` | Public front for [[systems/omniroute]] |
|
||||||
|
| `nanobot.martinwa.org` / `nano-ssh.martinwa.org` | `100.122.189.81` (ct333 "clawtest") | See [[comparisons/hermes-vs-nanobot]] |
|
||||||
|
|
||||||
|
`mcp-portal.martinwa.org` is not present in the ct207 ingress list captured during this audit — it may be served by a different tunnel/host, or the ingress list may have moved since. **Unknown**, flagged for follow-up rather than guessed.
|
||||||
|
|
||||||
|
## Full ct207 ingress list (context, not all Hermes-related)
|
||||||
|
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
||||||
|
|
||||||
|
## History
|
||||||
|
- ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/diagnose-docker]]
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/cloudflare-tunnel]]
|
||||||
|
- [[infrastructure/network-map]]
|
||||||
|
- [[infrastructure/hosts]]
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
---
|
||||||
|
title: Known Hosts and Paths
|
||||||
|
type: infrastructure
|
||||||
|
status: partial
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-14
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [infrastructure, hosts, containers, paths]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Known Hosts and Paths
|
||||||
|
|
||||||
|
## Verified Hosts
|
||||||
|
|
||||||
|
| Host | Role | Address | Verified |
|
||||||
|
|---|---|---|---|
|
||||||
|
| localhost (this host) | Primary Hermes runtime, Xvfb, file storage | `:102` display | Yes |
|
||||||
|
| litellm | LiteLLM proxy | `http://litellm:4000/v1` | Yes (from config) |
|
||||||
|
| omni | OmniRoute proxy | `http://omni:20128/v1` | Yes (from config, 2026-08-13) |
|
||||||
|
| CT201 | Steel Browser, Runtipi, Homepage dashboard | `100.96.244.39` (Tailscale), `192.168.178.166` (LAN) | Yes (live probe) |
|
||||||
|
| CT460 | Hermes host, Camoufox, cua-driver | `hermes.kangaroo-eel.ts.net` | Yes |
|
||||||
|
| CT450 (hermesagent) | Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite) | `100.98.205.112` (Tailscale), `192.168.178.65` (LAN) | Yes |
|
||||||
|
|
||||||
|
## Project Paths
|
||||||
|
|
||||||
|
| Path | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `/home/hermes` | Hermes home | `~/.hermes/` contains config, skills, cache |
|
||||||
|
| `/home/hermes/workspace` | Workspace | default terminal cwd |
|
||||||
|
| `/home/hermes/.local/bin/hermes` | Hermes binary | |
|
||||||
|
| `/home/hermes/.hermes/config.yaml` | Config | version 33 |
|
||||||
|
| `/home/hermes/.hermes/skills/` | Skills dir | 66 skills installed |
|
||||||
|
| `/home/hermes/.hermes/cron/` | Cron jobs | 25 active jobs |
|
||||||
|
| `/home/hermes/.hermes/mnemosyne/` | Mnemosyne DB | backups/, data/, logs/, models/ |
|
||||||
|
| `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver` | cua-driver | v0.9.0 |
|
||||||
|
| `~/.camofox/` | Camofox profiles | managed persistence |
|
||||||
|
| `~/.camoufox/` | Camoufox profiles | multiple profiles present |
|
||||||
|
| `/home/hermes/.hermes/browser_screenshots/` | Browser captures | |
|
||||||
|
| `/home/hermes/wiki/` | Wiki | This knowledge base |
|
||||||
|
| `/home/hermes/obsidian-vault/` | Obsidian vault | personal notes (50 - Research etc.) |
|
||||||
|
| `/home/hermes/hermes-webui/` | Hermes WebUI | web UI source |
|
||||||
|
| `/home/hermes/hermes-ui/` | Hermes Desktop browser (Vite dev) | present on CT460 and CT450 port 5174 |
|
||||||
|
| `/home/hermes/camofox-browser/` | Camofox browser | |
|
||||||
|
| `/home/hermes/camoufox/` | Camoufox source | |
|
||||||
|
| `/home/hermes/services/` | Services dir | |
|
||||||
|
| `/home/hermes/www/` | Web root | |
|
||||||
|
| `/home/hermes/go/` | Go workspace | |
|
||||||
|
| `/home/hermes/Downloads/` | Downloads | |
|
||||||
|
|
||||||
|
## Compose Files (Docker runtime state unknown)
|
||||||
|
- `/home/hermes/.hermes/hermes-agent/docker-compose.yml`
|
||||||
|
- `/home/hermes/hermes-webui/docker-compose.yml`
|
||||||
|
- `/home/hermes/workspace/readlater/docker-compose.yml`
|
||||||
|
|
||||||
|
## Homelab References
|
||||||
|
- **PVE host:** `pve.kangaroo-eel.ts.net` (100.96.100.82) per session notes — Unverified on this host
|
||||||
|
- **CT600:** OmniRoute server (retired → CT601 `omni`, 2026-08-13) — Verified via `pct list`
|
||||||
|
|
||||||
|
## Unverified
|
||||||
|
- CT600, CT333, and other remote Proxmox containers/VMs referenced in session notes but not confirmed from this host
|
||||||
|
- PVE host identity not confirmed from this host without explicit confirmation
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[docker-services]]
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
---
|
||||||
|
title: Network Map
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [infrastructure, network]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Network Map
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Where Hermes and its dependencies run, and how traffic reaches them.
|
||||||
|
|
||||||
|
## Zones
|
||||||
|
|
||||||
|
### LAN
|
||||||
|
- **Cluster host (pve):** `192.168.178.39` (Proxmox VE, kernel 6.17.4-1-pve)
|
||||||
|
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
|
||||||
|
|
||||||
|
### Tailscale (`kangaroo-eel.ts.net`)
|
||||||
|
- **pve:** `100.96.100.82`
|
||||||
|
- **hermes (ct460):** `100.118.5.51`
|
||||||
|
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
|
||||||
|
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
|
||||||
|
- **nanobot (ct333, "clawtest"):** `100.122.189.81`
|
||||||
|
- **webserver (ct207, runs the Cloudflare Tunnel):** `100.82.40.23`
|
||||||
|
- **debianvm (VM 101):** `100.98.95.65`
|
||||||
|
- **docker host (ct200):** `100.96.150.69`
|
||||||
|
- **openweb (ct245):** `100.69.230.13`
|
||||||
|
- **nextcloud (ct270):** `100.90.102.19`
|
||||||
|
|
||||||
|
### Public (Cloudflare Tunnel)
|
||||||
|
- Origin daemon runs on **ct207 (webserver)**, not on this host. See [[systems/cloudflare-tunnel]] and [[infrastructure/domains-and-tunnels]] for the full ingress list.
|
||||||
|
- This host is reached publicly via two routes defined in that tunnel: `hermes.martinwa.org` → `100.118.5.51:19119` (nginx-fronted webui) and `hermes-desktop.martinwa.org` → `100.118.5.51:5174` (desktop companion).
|
||||||
|
|
||||||
|
## Local ports on this host (ct460)
|
||||||
|
| Port | Bind | Process | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 22 | `*` | sshd | SSH |
|
||||||
|
| 80 / [::]:80 | `*` | nginx | reverse proxy, default vhost |
|
||||||
|
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (tunneled) |
|
||||||
|
| 9119 | `0.0.0.0` | hermes (webui) | Hermes Web UI backend |
|
||||||
|
| 8644 / 8642 | `*` | hermes | additional Hermes agent ports |
|
||||||
|
| 5174 | `*` | node | Hermes desktop companion (tunneled as hermes-desktop.martinwa.org) |
|
||||||
|
| 9377 | `*` | node | Stratos — AI Command Centre |
|
||||||
|
| 8787 | `127.0.0.1`* | python | Hermes gateway health endpoint (see [[runbooks/update-hermes-safely]]) |
|
||||||
|
| 9099 / 8790 | `*` | python3 | auxiliary Hermes processes (unverified purpose) |
|
||||||
|
| 5901 / [::]:5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
|
||||||
|
| 6080 / 6081 | `127.0.0.1` | websockify | noVNC web bridges for two VNC displays |
|
||||||
|
| 25 | `127.0.0.1` / `[::1]` | postfix | local mail transport |
|
||||||
|
|
||||||
|
\* `curl` target used in [[runbooks/update-hermes-safely]] health check; binding not independently re-verified in this pass.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/diagnose-docker]]
|
||||||
|
- [[runbooks/recover-docker-service]]
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[infrastructure/hosts]]
|
||||||
|
- [[infrastructure/domains-and-tunnels]]
|
||||||
|
- [[systems/cloudflare-tunnel]]
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
title: Proxmox VE
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-26
|
||||||
|
updated: 2026-08-13
|
||||||
|
confidence: high
|
||||||
|
tags: [proxmox, virtualization, infrastructure, homelab]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Proxmox VE
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||||
|
|
||||||
|
## Details
|
||||||
|
- **Host:** `pve.kangaroo-eel.ts.net` (Tailscale: `100.96.100.82`)
|
||||||
|
- **Node name:** `pve`
|
||||||
|
- **Version:** Proxmox VE 8.x
|
||||||
|
- **Primary user:** `Anthony@pam` (full admin)
|
||||||
|
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
|
||||||
|
|
||||||
|
## VMs (QEMU)
|
||||||
|
| VMID | Name | Status | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 101 | debianvm | running | Debian VM — Portainer, filebrowser, code-server, Sunshine |
|
||||||
|
| 100 | homeassistant | running | Home Assistant |
|
||||||
|
| 282 | ubuntu-server-xfce | running | Ubuntu desktop — Sunshine, Kubuntu |
|
||||||
|
| 403 | open-claw | running | OpenClaw agent |
|
||||||
|
| 401 | hermes-old | stopped | Legacy Hermes VM (deprecated) |
|
||||||
|
|
||||||
|
## LXC Containers
|
||||||
|
| VMID | Name | Status | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 200 | docker | running | Main Docker host — 42+ containers |
|
||||||
|
| 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. |
|
||||||
|
| 205 | ollama | running | Ollama LLM inference + Guanaco proxy (v0.8.14, updated 2026-08-14) |
|
||||||
|
| 207 | webserver | running | Web server — Vercel AI, manifest router |
|
||||||
|
| 215 | n8n | running | N8n automation |
|
||||||
|
| 221 | gitea | running | Gitea git server |
|
||||||
|
| 241 | ntfy | running | Ntfy push notifications |
|
||||||
|
|| 245 | openweb | running | Open WebUI (Jul 30 updated, memory 4096 MB) |
|
||||||
|
| 255 | mattermost | running | Mattermost chat |
|
||||||
|
| 260 | freshrss | running | FreshRSS feed reader |
|
||||||
|
| 270 | nextcloud | running | NextCloud |
|
||||||
|
| 299 | sftpgo | running | SFTPGo file server — security review 2026-08-09: connects as root, unpinned fingerprint, `//` root path |
|
||||||
|
| 310 | monitoring | stopped | Monitoring stack (deprecated) |
|
||||||
|
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
|
||||||
|
| 450 | hermesagent | running | Hermes Agent LXC |
|
||||||
|
| 460 | hermes | running | Hermes CT460 |
|
||||||
|
| 465 | qwenpaw | running | QwenPaw LLM |
|
||||||
|
| 500 | litellm | running | LiteLLM proxy |
|
||||||
|
| 555 | elementsynapse | stopped | Element Synapse (deprecated) |
|
||||||
|
| 600 | omniroute | stopped | OmniRoute model router — retired 2026-08-13, migrated to CT601 |
|
||||||
|
| 601 | omni | running | OmniRoute model router — successor to CT600, since 2026-08-13 |
|
||||||
|
| 103 | cliproxyapi | running | Cliproxy API |
|
||||||
|
| 104 | rclone | running | Rclone sync |
|
||||||
|
| 106 | glance | stopped | Glance dashboard |
|
||||||
|
| 209 | cloudreve | stopped | Cloudreve (deprecated) |
|
||||||
|
|
||||||
|
## API Tokens
|
||||||
|
- `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard)
|
||||||
|
|
||||||
|
## Key Paths
|
||||||
|
- `/etc/pve/` — PVE config directory
|
||||||
|
- `/var/lib/vz/` — Local storage
|
||||||
|
- `/etc/pve/priv/token/` — API token storage
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/homepage-dashboard]] — Dashboard shows PVE stats
|
||||||
|
- [[infrastructure/hosts]] — Full host inventory
|
||||||
|
- [[infrastructure/network-map]] — Network topology
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
title: Tailscale
|
||||||
|
type: infrastructure
|
||||||
|
status: active
|
||||||
|
created: 2026-07-26
|
||||||
|
updated: 2026-07-27
|
||||||
|
verified_on: 2026-07-27
|
||||||
|
confidence: high
|
||||||
|
tags: [tailscale, vpn, networking, infrastructure]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Tailscale
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access via Tailscale Serve.
|
||||||
|
|
||||||
|
## Domain
|
||||||
|
- **Tailnet:** `kangaroo-eel.ts.net`
|
||||||
|
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`)
|
||||||
|
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||||
|
|
||||||
|
## CT460 Serve Routes
|
||||||
|
| Path | Target | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `/` | `http://127.0.0.1:9119` | Hermes WebUI |
|
||||||
|
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
|
||||||
|
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
||||||
|
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
||||||
|
|
||||||
|
## CT460 Funnel Routes
|
||||||
|
| Port | Target | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel |
|
||||||
|
|
||||||
|
## CT201 Serve Routes
|
||||||
|
| Path | Target | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `:8443` | `http://localhost:8082` | Homepage dashboard |
|
||||||
|
| `/steel` | `http://127.0.0.1:5173` | Steel Browser UI |
|
||||||
|
| `/steel-api` | `http://127.0.0.1:3000` | Steel Browser API (path prefix not stripped — use direct IP for API calls) |
|
||||||
|
|
||||||
|
## Atomic Restore Script
|
||||||
|
`/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` — Rebuilds the complete CT460 route table, including tailnet Serve on `:443` and public Funnel on `:8443`.
|
||||||
|
|
||||||
|
`tailscale serve reset` clears Funnel state as well as Serve state. The atomic script therefore restores `/webhook` after rebuilding the 443 routes. Do not use the older Serve-only restore path for complete recovery.
|
||||||
|
|
||||||
|
## Key Commands
|
||||||
|
```bash
|
||||||
|
# Check routes
|
||||||
|
tailscale serve status
|
||||||
|
|
||||||
|
# Rebuild routes (CT460)
|
||||||
|
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
|
||||||
|
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
|
||||||
|
|
||||||
|
# Add route
|
||||||
|
tailscale serve --set-path /jobs http://127.0.0.1:9099
|
||||||
|
|
||||||
|
# Remove route
|
||||||
|
tailscale serve --set-path /jobs off
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there
|
||||||
|
- CT460 has no conflict — Tailscale uses 443 directly
|
||||||
|
- Routes must be re-added in order (restore script handles this)
|
||||||
|
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[infrastructure/domains-and-tunnels]] — Full DNS and tunnel inventory
|
||||||
|
- [[systems/homepage-dashboard]] — Homepage on CT201 via Tailscale
|
||||||
|
- [[runbooks/job-radar]] — Dashboard incident and route verification
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
# Wiki Log
|
||||||
|
|
||||||
|
> Chronological record of wiki actions. Append-only.
|
||||||
|
|
||||||
|
## [2026-07-22] init | Fresh Hermes Systems Wiki
|
||||||
|
- Created technical wiki at `/home/hermes/wiki`
|
||||||
|
- Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
|
||||||
|
- Wrote SCHEMA.md with 11 operating rules
|
||||||
|
- Wrote index.md as curated Start here dashboard
|
||||||
|
- Reseeded system/infrastructure/runbook stubs for existing homelab
|
||||||
|
- Git initialized, local-only
|
||||||
|
- No secrets, passwords, tokens, or private keys written to vault
|
||||||
|
|
||||||
|
## [2026-07-22] audit | External audit pass (Claude, at Anthony's request)
|
||||||
|
- Merged duplicate pages: `systems/omniRoute.md` (empty stub) -> kept `systems/omniroute.md` (populated); repointed 5 links
|
||||||
|
- Merged duplicate pages: `runbooks/safely-update-hermes.md` -> kept as `runbooks/update-hermes-safely.md` (fuller content); repointed 2 links
|
||||||
|
- Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
|
||||||
|
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
|
||||||
|
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
|
||||||
|
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
|
||||||
|
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
|
||||||
|
|
||||||
|
## [2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
|
||||||
|
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
|
||||||
|
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
|
||||||
|
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
|
||||||
|
- Fixed a broken `review_after: 0001-01-01` in decisions/keep-openrouter-as-fallback.md
|
||||||
|
- New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
|
||||||
|
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
|
||||||
|
|
||||||
|
## [2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
|
||||||
|
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
|
||||||
|
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
|
||||||
|
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
|
||||||
|
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
|
||||||
|
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
|
||||||
|
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
|
||||||
|
|
||||||
|
## [2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
|
||||||
|
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
|
||||||
|
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
|
||||||
|
|
||||||
|
## [2026-07-25] operations | Tool Search and gateway resilience
|
||||||
|
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
|
||||||
|
- Documented how new skill categories are derived from the first folder below `~/.hermes/skills/`
|
||||||
|
- Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
|
||||||
|
- Recorded the I/O-controller delegation limitation so an ineffective `IOWeight` is not mistaken for active protection
|
||||||
|
- Added health checks, exact effective settings, backup location, and rollback procedure
|
||||||
|
- No secrets, tokens, or private credentials were written to the wiki
|
||||||
|
|
||||||
|
## [2026-07-28] dashboard | Jobs Dashboard feature additions
|
||||||
|
- Fixed bulk status update (payload shape mismatch: `{ids, status}` → `{ids, updates: {status}}`)
|
||||||
|
- Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
|
||||||
|
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
|
||||||
|
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
|
||||||
|
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion `Notes / JD Summary`
|
||||||
|
- Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
|
||||||
|
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
|
||||||
|
- Updated Synergy and Dept of Communities to Phone Screen status
|
||||||
|
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
|
||||||
|
|
||||||
|
## [2026-07-27] incident | Job Radar search and dashboard recovery
|
||||||
|
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
|
||||||
|
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require `MISSING_COVER_LETTER_COUNT=0`
|
||||||
|
- Recorded the production cron model (`gpt-5.6-terra`) and verified first inherited fallback (`opencode-go/mimo-v2.5`)
|
||||||
|
- Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
|
||||||
|
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both `:443` Serve and `:8443/webhook` Funnel
|
||||||
|
- Added [[runbooks/job-radar]] and updated scheduled-task, Tailscale and index references
|
||||||
|
- Verified no secrets, API keys or private credentials were written to the wiki
|
||||||
|
|
||||||
|
## [2026-07-29] update | Steel Browser deployment — browser backend switch
|
||||||
|
- **New page:** `systems/browser-backend.md` — rewritten to reflect Steel Browser as primary, Camoufox as fallback
|
||||||
|
- **New page:** `comparisons/steel-vs-camoufox.md` — comparison of the two backends
|
||||||
|
- **Updated:** `infrastructure/hosts.md` — added CT201 and CT460 to verified hosts table
|
||||||
|
- **Updated:** `infrastructure/docker-services.md` — added Steel Browser compose file
|
||||||
|
- **Updated:** `infrastructure/tailscale.md` — added CT201 Steel serve routes
|
||||||
|
- **Updated:** `index.md` — browser-backend link description
|
||||||
|
- **Updated:** `current-state.md` — added 2026-07-29 changes section
|
||||||
|
- **Config change:** `browser.cdp_url` set to `http://100.96.244.39:9223` in ops profile
|
||||||
|
|
||||||
|
## [2026-07-30] update | Open WebUI updated on CT245
|
||||||
|
- Updated image to `ghcr.io/open-webui/open-webui:main` (new digest e97bf9531916, Jul 27)
|
||||||
|
- Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
|
||||||
|
- Restarted container after OOM hang
|
||||||
|
- Updated `infrastructure/proxmox.md`
|
||||||
|
|
||||||
|
## [2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
|
||||||
|
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
|
||||||
|
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
|
||||||
|
- Ran `mnemosyne reindex --yes` to rebuild all embeddings — working memory now at 100%
|
||||||
|
- Updated health check script (`mnemosyne-health-check.sh`) to report actual embedding coverage via new helper script (`check_embedding_coverage.py`)
|
||||||
|
- Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
|
||||||
|
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
|
||||||
|
- DB grew from 8.7M to 21M with the new embeddings
|
||||||
|
|
||||||
|
## [2026-08-01] incident | Mnemosyne capture pipeline repaired
|
||||||
|
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no `working_memory` row existed.
|
||||||
|
- Corrected stringified structured YAML (`sync_roles`, `skip_contexts`, `reflect`, `sleep_threshold`, `ignore_patterns`) and enabled profile isolation; `hermes config check` passed.
|
||||||
|
- Verified the repair with a second CLI marker that persisted as an exact normalized `[USER]` database row.
|
||||||
|
- Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route `:9119`; added authoritative identity/preference/workflow slots.
|
||||||
|
- Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
|
||||||
|
- Added [[incidents/2026-08-01-mnemosyne-capture-failure]] and refreshed [[systems/mnemosyne]] plus [[systems/memory-backend]].
|
||||||
|
|
||||||
|
## [2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
|
||||||
|
- Rewrote [[comparisons/steel-vs-camoufox]] — full Steel experiment story, timeline, lessons learned
|
||||||
|
- Rewrote [[systems/browser-backend]] — Camoufox primary, Steel disabled
|
||||||
|
- Updated [[current-state]] — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
|
||||||
|
- Updated [[concepts/browser-automation]] — added Steel experiment lesson
|
||||||
|
- Created [[decisions/rollback-steel-to-camofox]] — formal decision record
|
||||||
|
- Updated [[systems/scheduled-tasks]] — job count 25, Uber Eats upgrade, GH Trending Daily removed
|
||||||
|
- Updated [[systems/mnemosyne]] — vector embedding coverage gap documented
|
||||||
|
- Updated [[systems/model-providers]] — DeepSeek V4 Flash 0731 details
|
||||||
|
- Created [[reports/uber-eats/index]] — landing page for monthly spend reports
|
||||||
|
- Updated [[decisions/index]] — linked new decision page
|
||||||
|
- Updated [[index.md]] — browser description, reports section, last-updated date
|
||||||
|
- Created nightly wiki audit cron job (23:59 AWST)
|
||||||
|
|
||||||
|
## [2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
|
||||||
|
- Ops profile (`~/.hermes/profiles/ops/config.yaml`): enabled `mnemosyne` plugin with `profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|
||||||
|
- Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
|
||||||
|
- Verified end-to-end: memory writes land correctly in both profiles independently
|
||||||
|
- Updated [[systems/mnemosyne]] — profile isolation section now documents ops config
|
||||||
|
- Updated [[current-state]] — added ops profile Mnemosyne entry
|
||||||
|
- Updated [[runbooks/multiplexer-setup]] — fixed "Elephant ops" → "Tusk ops" in architecture diagram
|
||||||
|
- Fixed [[systems/memory-backend]] — "Shared surface empty" section renamed to reflect intentional isolation decision
|
||||||
|
|
||||||
|
## [2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
|
||||||
|
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
|
||||||
|
- Reworked `deepseek-flash-free` and `good-free` on CT600: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
|
||||||
|
- Created [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
|
||||||
|
- Updated [[current-state]] — added OmniRoute free-DeepSeek mitigation to Recent Changes
|
||||||
|
- Updated [[systems/omniroute]] — added Recent Changes (2026-08-04) section
|
||||||
|
- Updated [[systems/model-providers]] — added mitigation entry under Notable Model Updates, updated `updated` date
|
||||||
|
|
||||||
|
## [2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
|
||||||
|
- **OmniRoute three-layer fix documented:** npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. `is_active=0` does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
|
||||||
|
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-07) section with full root cause analysis
|
||||||
|
- **Updated [[systems/nanobot]]** — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
|
||||||
|
- **Updated [[current-state]]** — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
|
||||||
|
- **Rewrote [[systems/scheduled-tasks]]** — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
|
||||||
|
- **Wiki audit cron error noted:** model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
|
||||||
|
- **Updated [[index.md]]** — last-updated date 2026-08-04 → 2026-08-07
|
||||||
|
|
||||||
|
## [2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
|
||||||
|
- **Portainer password reset documented:** CT245 admin user confirmed as `anthony` (not `admin`). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to [[current-state]].
|
||||||
|
- **SFTPGo security review documented:** CT299 findings — root access, unpinned fingerprint, `//` root path, F1 quota mystery. Added to [[current-state]] and [[infrastructure/proxmox]].
|
||||||
|
|
||||||
|
## [2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
|
||||||
|
- **Cron job model repinning:** 13 jobs repinned from dead/stale provider combos. 3 from `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker from `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named jobs (e.g. `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`) → `smart-route`. Daily Marketing Job Radar → `gpt-5.6-terra`.
|
||||||
|
- **Hermes UI vite config revert:** Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
|
||||||
|
- **Phase One estate review completed:** Manual cron run (ID `a9359b7db27b`) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, CT333 Nanobot has live `jobs.json` with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
|
||||||
|
- **Updated [[current-state]]** — added 2026-08-08 changes section
|
||||||
|
- **Updated [[systems/scheduled-tasks]]** — Phase One review marked completed, model repinning noted in Known Issues
|
||||||
|
|
||||||
|
## [2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
|
||||||
|
- **Cron job failures documented:** Four jobs failed on 2026-08-10 with two error patterns:
|
||||||
|
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
|
||||||
|
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
|
||||||
|
- **Root cause identified:** Disk at 83% (11GB free on 65GB), `state.db` at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
|
||||||
|
- **Updated [[current-state]]** — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
|
||||||
|
- **Updated [[systems/scheduled-tasks]]** — added session storage failures to Known Issues, updated verified_on to 2026-08-10
|
||||||
|
- **OpenCode WebSocket proxy errors noted:** Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
|
||||||
|
- **No new pages created** — all changes applied to existing pages
|
||||||
|
|
||||||
|
## [2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
|
||||||
|
- **OmniRoute AutoRoute variants documented:** `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: `auto/best-free` routed to Antigravity. Updated [[systems/omniroute]].
|
||||||
|
- **OpenViking memory provider evaluated, NOT adopted:** single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains `memory.provider: mnemosyne`. Added to [[comparisons/memory-backends]].
|
||||||
|
- **FreeLLMAPI router internals documented:** 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default `freellmapi-auto`. Added to [[comparisons/model-routers]].
|
||||||
|
- **No new pages created** — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.
|
||||||
|
|
||||||
|
## [2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
|
||||||
|
- **OmniRoute migrated from CT600 to CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. All facts verified live (`pct list`, `grep`, Tailscale DNS).
|
||||||
|
- **Updated [[systems/omniroute]]** — Current Configuration rewritten for `omni`/CT601, added Recent Changes (2026-08-13), bumped `updated`/`verified_on` to 2026-08-13.
|
||||||
|
- **Updated [[infrastructure/proxmox]]** — CT600 marked stopped/retired, CT601 added running; bumped `updated`.
|
||||||
|
- **Updated [[current-state]]** — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
|
||||||
|
- **Updated [[infrastructure/hosts]]** — OmniRoute proxy entry → `omni`; CT600 reference marked verified/retired.
|
||||||
|
- **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13.
|
||||||
|
- No new pages created; all changes applied to existing pages. No secrets written.
|
||||||
|
|
||||||
|
## [2026-08-14] update | Nightly wiki audit — OmniRoute factory reset, Guanaco update, GPT Luna research
|
||||||
|
- **OmniRoute factory reset documented:** CT601 database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys invalid — removed. Multiple free combos broken by upstream model ID changes (OpenRouter Poolside dash→dot, Groq llama-4-scout retired). Default model changed from `auto/best-free` to `resilient-free`. Working free combos: `free-lkgp`, `mega-free`, `smart-route`. See [[systems/omniroute]].
|
||||||
|
- **Guanaco router documented:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. Created [[systems/guanaco]].
|
||||||
|
- **GPT-5.6 Luna research documented:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna for planning/review, DeepSeek for execution. Updated [[systems/model-providers]].
|
||||||
|
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-14), bumped `updated`/`verified_on`
|
||||||
|
- **Updated [[systems/model-providers]]** — provider table (`omniroute` → `omni`), Guanaco entry, GPT-5.6 Luna entry, default model changed to `resilient-free`, bumped dates
|
||||||
|
- **Created [[systems/guanaco]]** — new page for Guanaco LLM proxy on CT205
|
||||||
|
- **Updated [[current-state]]** — added Recent Changes (2026-08-14) with all seven items, bumped `updated`
|
||||||
|
- **Updated [[infrastructure/proxmox]]** — CT205 description updated with Guanaco proxy
|
||||||
|
- **Updated [[infrastructure/hosts]]** — bumped `updated` date
|
||||||
|
- **Updated [[index]]** — last-updated date → 2026-08-14, added Guanaco link to Core systems
|
||||||
|
- No secrets written.
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
---
|
||||||
|
title: Retained Research Answers
|
||||||
|
type: query
|
||||||
|
status: unresolved
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
review_after: 2026-08-22
|
||||||
|
confidence: low
|
||||||
|
tags: [query, research]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
||||||
|
|
||||||
|
# Retained Research Answers
|
||||||
|
Filed research answers worth reusing.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,87 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Uber Eats Monthly Spend — July 2026</title>
|
||||||
|
<style>
|
||||||
|
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||||
|
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f8f9fa; color: #1a1a2e; padding: 40px 20px; }
|
||||||
|
.card { max-width: 640px; margin: 0 auto; background: #fff; border-radius: 20px; padding: 32px; box-shadow: 0 2px 12px rgba(0,0,0,0.06); }
|
||||||
|
.header { display: flex; align-items: center; gap: 14px; margin-bottom: 28px; }
|
||||||
|
.icon { font-size: 36px; line-height: 1; }
|
||||||
|
h1 { font-size: 22px; font-weight: 700; color: #1a1a2e; }
|
||||||
|
.subtitle { font-size: 13px; color: #6b7280; margin-top: 2px; }
|
||||||
|
.summary-row { display: flex; justify-content: space-between; gap: 12px; margin-bottom: 28px; }
|
||||||
|
.stat { flex: 1; background: #f3f4f6; border-radius: 14px; padding: 16px; text-align: center; }
|
||||||
|
.stat-value { font-size: 22px; font-weight: 700; color: #1a1a2e; }
|
||||||
|
.stat-label { font-size: 11px; color: #6b7280; margin-top: 2px; text-transform: uppercase; letter-spacing: 0.5px; }
|
||||||
|
.stat.up .stat-value { color: #dc2626; }
|
||||||
|
.section-title { font-size: 14px; font-weight: 600; color: #374151; margin: 24px 0 12px; padding-bottom: 8px; border-bottom: 1px solid #e5e7eb; }
|
||||||
|
.order-item { display: flex; justify-content: space-between; align-items: center; padding: 10px 0; border-bottom: 1px solid #f3f4f6; }
|
||||||
|
.order-item:last-child { border-bottom: none; }
|
||||||
|
.order-left { display: flex; flex-direction: column; }
|
||||||
|
.order-restaurant { font-size: 14px; font-weight: 600; color: #1a1a2e; }
|
||||||
|
.order-date { font-size: 12px; color: #9ca3af; }
|
||||||
|
.order-amount { font-size: 15px; font-weight: 700; color: #1a1a2e; }
|
||||||
|
.freq-item { display: flex; justify-content: space-between; padding: 8px 0; font-size: 14px; }
|
||||||
|
.freq-name { color: #1a1a2e; }
|
||||||
|
.freq-count { color: #6b7280; font-weight: 500; }
|
||||||
|
.vs { font-size: 12px; color: #9ca3af; padding: 8px 0 0; }
|
||||||
|
.vs span { color: #374151; font-weight: 500; }
|
||||||
|
.footer { font-size: 11px; color: #9ca3af; margin-top: 24px; text-align: center; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="card">
|
||||||
|
<div class="header">
|
||||||
|
<div class="icon">🍔</div>
|
||||||
|
<div>
|
||||||
|
<h1>Uber Eats — July 2026</h1>
|
||||||
|
<div class="subtitle">Monthly spend report • 01 Aug 2026</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="summary-row">
|
||||||
|
<div class="stat">
|
||||||
|
<div class="stat-value">$264.51</div>
|
||||||
|
<div class="stat-label">Total Spend</div>
|
||||||
|
</div>
|
||||||
|
<div class="stat">
|
||||||
|
<div class="stat-value">11</div>
|
||||||
|
<div class="stat-label">Orders</div>
|
||||||
|
</div>
|
||||||
|
<div class="stat">
|
||||||
|
<div class="stat-value">$24.05</div>
|
||||||
|
<div class="stat-label">Avg/Order</div>
|
||||||
|
</div>
|
||||||
|
<div class="stat up">
|
||||||
|
<div class="stat-value">+124%</div>
|
||||||
|
<div class="stat-label">vs 2026-06</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="section-title">🍽️ Top 5 Most Expensive</div>
|
||||||
|
<div class="order-item"><div class="order-left"><span class="order-restaurant">Coles</span><span class="order-date">2026-07-11 17:02</span></div><div class="order-amount">$40.73</div></div>
|
||||||
|
<div class="order-item"><div class="order-left"><span class="order-restaurant">Foodies Market IGA</span><span class="order-date">2026-07-09 15:42</span></div><div class="order-amount">$36.35</div></div>
|
||||||
|
<div class="order-item"><div class="order-left"><span class="order-restaurant">IGA Grocery</span><span class="order-date">2026-07-23 20:31</span></div><div class="order-amount">$34.38</div></div>
|
||||||
|
<div class="order-item"><div class="order-left"><span class="order-restaurant">806 Noodle House</span><span class="order-date">2026-07-06 14:19</span></div><div class="order-amount">$24.66</div></div>
|
||||||
|
<div class="order-item"><div class="order-left"><span class="order-restaurant">Lepak Kopitiam Vic Park, also known as Yum Yum Tree Vic Park</span><span class="order-date">2026-07-09 12:56</span></div><div class="order-amount">$24.28</div></div>
|
||||||
|
|
||||||
|
|
||||||
|
<div class="section-title">🔄 Most Frequented</div>
|
||||||
|
<div class="freq-item"><span class="freq-name">Hungry Jacks</span><span class="freq-count">3 orders</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">Angelo Street Market</span><span class="freq-count">2 orders</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">806 Noodle House</span><span class="freq-count">2 orders</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">IGA Grocery</span><span class="freq-count">1 order</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">Coles</span><span class="freq-count">1 order</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">Foodies Market IGA</span><span class="freq-count">1 order</span></div>
|
||||||
|
<div class="freq-item"><span class="freq-name">Lepak Kopitiam Vic Park</span><span class="freq-count">1 order</span></div>
|
||||||
|
|
||||||
|
|
||||||
|
<div class="vs">vs 2026-06: <span>+124%</span> in spend (+$146.46) • orders went from 3 to 11</div>
|
||||||
|
|
||||||
|
<div class="footer">Generated from Camofox authenticated Uber Eats session</div>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# 🍔 Uber Eats Monthly Spend — 2026-07
|
||||||
|
|
||||||
|
Generated: 2026-08-01T03:30:00+08:00
|
||||||
|
Source: Camofox authenticated Uber Eats session (profile anthony) via getPastOrdersV1 API
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|---|---|
|
||||||
|
| Total Spend | $264.51 AUD |
|
||||||
|
| Total Orders | 11 |
|
||||||
|
| Avg per Order | $24.05 |
|
||||||
|
| Most Expensive | $40.73 |
|
||||||
|
| Least Expensive | $0.00 |
|
||||||
|
|
||||||
|
## vs Previous Month (2026-06)
|
||||||
|
|
||||||
|
| | This Month | Last Month | Change |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Spend | $264.51 | $118.05 | +$146.46 |
|
||||||
|
| Orders | 11 | 3 | +8 |
|
||||||
|
| Avg/Order | $24.05 | $39.35 | $-15.30 |
|
||||||
|
|
||||||
|
## Top 5 Most Expensive Orders
|
||||||
|
|
||||||
|
| # | Date | Restaurant | Amount |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | 2026-07-11 17:02 | Coles | $40.73 |
|
||||||
|
| 2 | 2026-07-09 15:42 | Foodies Market IGA | $36.35 |
|
||||||
|
| 3 | 2026-07-23 20:31 | IGA Grocery | $34.38 |
|
||||||
|
| 4 | 2026-07-06 14:19 | 806 Noodle House | $24.66 |
|
||||||
|
| 5 | 2026-07-09 12:56 | Lepak Kopitiam Vic Park, also known as Yum Yum Tree Vic Park | $24.28 |
|
||||||
|
|
||||||
|
## Most Frequented Restaurants
|
||||||
|
|
||||||
|
| Restaurant | Orders |
|
||||||
|
|---|---|
|
||||||
|
| Hungry Jacks | 3 |
|
||||||
|
| Angelo Street Market | 2 |
|
||||||
|
| 806 Noodle House | 2 |
|
||||||
|
| IGA Grocery | 1 |
|
||||||
|
| Coles | 1 |
|
||||||
|
| Foodies Market IGA | 1 |
|
||||||
|
| Lepak Kopitiam Vic Park | 1 |
|
||||||
|
|
||||||
|
## All Orders
|
||||||
|
|
||||||
|
| Date | Time | Restaurant | Amount |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 2026-07-03 | 19:16 | Hungry Jacks | $21.11 |
|
||||||
|
| 2026-07-06 | 14:19 | 806 Noodle House | $24.66 |
|
||||||
|
| 2026-07-08 | 14:00 | Angelo Street Market | $24.07 |
|
||||||
|
| 2026-07-09 | 12:56 | Lepak Kopitiam Vic Park, also known as Yum Yum Tree Vic Park | $24.28 |
|
||||||
|
| 2026-07-09 | 15:42 | Foodies Market IGA | $36.35 |
|
||||||
|
| 2026-07-11 | 17:02 | Coles | $40.73 |
|
||||||
|
| 2026-07-14 | 14:35 | Hungry Jacks | $11.20 |
|
||||||
|
| 2026-07-16 | 15:28 | 806 Noodle House | $23.66 |
|
||||||
|
| 2026-07-20 | 14:58 | Angelo Street Market | $24.07 |
|
||||||
|
| 2026-07-23 | 20:31 | IGA Grocery | $34.38 |
|
||||||
|
| 2026-07-25 | 12:58 | Hungry Jacks | $0.00 |
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
---
|
||||||
|
title: Uber Eats Monthly Reports
|
||||||
|
type: collection
|
||||||
|
status: active
|
||||||
|
created: 2026-08-02
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [reports, uber-eats, finance]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Uber Eats Monthly Reports
|
||||||
|
|
||||||
|
Monthly spend reports generated from Camofox-authenticated Uber Eats sessions.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
- Cron job runs on the 1st of each month at 02:00 AWST
|
||||||
|
- Opens a Camofox tab (profile `anthony`) and navigates to `ubereats.com/orders`
|
||||||
|
- Captures the `getPastOrdersV1` API response
|
||||||
|
- Saves JSON to `~/.hermes/uber_eats_report.json`
|
||||||
|
- Generates HTML report + wiki markdown page
|
||||||
|
- Delivers a formatted Telegram digest
|
||||||
|
|
||||||
|
## Reports
|
||||||
|
|
||||||
|
| Month | Spend | Orders | Avg/Order | Report |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 2026-07 | $264.51 | 11 | $24.05 | [[reports/uber-eats/2026-07]] |
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/scheduled-tasks]]
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
---
|
||||||
|
title: Backup Wiki
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, wiki, backup]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Backup Wiki
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Create a durable backup of the wiki so recoverable copies exist before mutating changes.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Before a big restructuring, page rewrite, or history migration
|
||||||
|
- Weekly or scheduled backup reminder
|
||||||
|
- Suspected accidental deletion or corruption
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Wiki path: `/home/hermes/wiki`
|
||||||
|
- Git is available
|
||||||
|
- Backup destination is writable
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Sanity-check repo state
|
||||||
|
```bash
|
||||||
|
cd /home/hermes/wiki && git status --short && git log --oneline -3
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Commit any open changes first
|
||||||
|
```bash
|
||||||
|
cd /home/hermes/wiki
|
||||||
|
git add -A
|
||||||
|
git commit -m "backup-baseline: $(date +%Y-%m-%d)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Export a timestamped copy outside the repo
|
||||||
|
```bash
|
||||||
|
BACKUP_DIR=/home/hermes/wiki-backups
|
||||||
|
mkdir -p "$BACKUP_DIR"
|
||||||
|
BACKUP="$BACKUP_DIR/wiki-$(date +%Y%m%d-%H%M%S).tar.gz"
|
||||||
|
tar -czf "$BACKUP" -C /home/hermes wiki
|
||||||
|
echo "Wrote $BACKUP"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Optional: push to a Git remote if configured
|
||||||
|
```bash
|
||||||
|
cd /home/hermes/wiki
|
||||||
|
git remote -v
|
||||||
|
git push --all
|
||||||
|
git push --tags
|
||||||
|
```
|
||||||
|
If no remote is configured, skip this step and rely on the tarball.
|
||||||
|
|
||||||
|
### 5. Rotate old backups if needed
|
||||||
|
Keep N most recent backups:
|
||||||
|
```bash
|
||||||
|
ls -1t "$BACKUP_DIR"/wiki-*.tar.gz | tail -n +6 | xargs -r rm
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `git status --short` shows a clean tree
|
||||||
|
- Tarball exists and is non-empty:
|
||||||
|
```bash
|
||||||
|
stat "$BACKUP"
|
||||||
|
tar -tzf "$BACKUP" | head -20
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- Restore from tarball:
|
||||||
|
```bash
|
||||||
|
BACKUP=$(ls -1t /home/hermes/wiki-backups/wiki-*.tar.gz | head -1)
|
||||||
|
rm -rf /home/hermes/wiki-restore
|
||||||
|
mkdir -p /home/hermes/wiki-restore
|
||||||
|
tar -xzf "$BACKUP" -C /home/hermes wiki-restore --strip-components=1
|
||||||
|
```
|
||||||
|
- Verify restored content before replacing live wiki
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- This runbook intentionally does not write secrets or token dumps into backups
|
||||||
|
- If you need a full Hermes config+data backup, use `hermes backup` in addition to this wiki-only backup
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[runbooks/update-hermes-safely]]
|
||||||
|
- [[current-state]]
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
title: Diagnose Failed Docker Service
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [runbook, docker, homelab]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Diagnose Failed Docker Service
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Diagnose container or compose issues on hosts where Docker is present.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Service reports container exit/failed state
|
||||||
|
- `docker ps` does not show an expected container
|
||||||
|
- Compose stack reports unhealthy status
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- `docker` installed and accessible to the user
|
||||||
|
- Docker daemon running
|
||||||
|
- Compose file path known for the affected project
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Check Docker daemon state
|
||||||
|
```bash
|
||||||
|
systemctl status docker
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. List containers
|
||||||
|
```bash
|
||||||
|
docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Inspect logs for the failing container
|
||||||
|
```bash
|
||||||
|
docker logs --tail 100 <container-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. If using compose, check compose state
|
||||||
|
```bash
|
||||||
|
cd /home/hermes/<project>/docker-compose
|
||||||
|
docker compose ps
|
||||||
|
docker compose logs --tail 100 <service-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Restart the failed container only
|
||||||
|
```bash
|
||||||
|
docker restart <container-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. If restart fails, investigate restart policy and image
|
||||||
|
```bash
|
||||||
|
docker inspect --format "{{.HostConfig.RestartPolicy}}" <container-name>
|
||||||
|
docker inspect --format "{{.Config.Image}}" <container-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `docker ps` shows the container `Up` or `healthy`
|
||||||
|
- Service or WebUI dependent on the container is reachable again
|
||||||
|
- Logs show clean startup messages
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- Do not delete volumes unless confirmed not in use
|
||||||
|
- If a container recreate is required, preserve volume mappings and env files
|
||||||
|
- If uncertain, collect logs first and ask before replacement
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- This runbook is generic. This current Hermes host does not have Docker installed as of 2026-07-22
|
||||||
|
- Docker-related paths must be confirmed before executing compose commands
|
||||||
|
- Do not invent container names or project paths
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22 on system where Docker runtime is absent; read-only steps verified, restart steps not executed
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[docker-services]]
|
||||||
|
- [[hosts]]
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
---
|
||||||
|
title: Hermes Gateway Resource Resilience
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-25
|
||||||
|
updated: 2026-07-25
|
||||||
|
review_after: 2026-10-23
|
||||||
|
verified_on: 2026-07-25
|
||||||
|
version_applies_to: Proxmox CT460
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, infrastructure, runbook]
|
||||||
|
sources: [live systemd and cgroup v2 state on CT460]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes Gateway Resource Resilience
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Keep Telegram and the Hermes gateway available when the browser or Web UI
|
||||||
|
causes severe memory pressure inside CT460.
|
||||||
|
|
||||||
|
The observed failure was real: the CT460 memory cgroup OOM-killed a large Web
|
||||||
|
UI process and a Chrome child. Because the gateway previously used
|
||||||
|
`OOMPolicy=stop`, the Chrome child death caused systemd to stop the entire
|
||||||
|
gateway until `Restart=always` brought it back.
|
||||||
|
|
||||||
|
## Active controls
|
||||||
|
|
||||||
|
### Gateway
|
||||||
|
|
||||||
|
File:
|
||||||
|
`/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf`
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Service]
|
||||||
|
OOMPolicy=continue
|
||||||
|
MemoryLow=512M
|
||||||
|
CPUWeight=1000
|
||||||
|
```
|
||||||
|
|
||||||
|
The gateway has no `MemoryHigh` or `MemoryMax`. It receives reclaim protection
|
||||||
|
and maximum relative CPU weight, while `OOMPolicy=continue` prevents an
|
||||||
|
OOM-killed browser child from stopping the whole unit. If the main gateway
|
||||||
|
process exits, its existing `Restart=always` policy still restarts it.
|
||||||
|
|
||||||
|
### Camofox
|
||||||
|
|
||||||
|
File:
|
||||||
|
`/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Service]
|
||||||
|
MemoryHigh=1G
|
||||||
|
MemoryMax=1536M
|
||||||
|
MemorySwapMax=768M
|
||||||
|
OOMPolicy=kill
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hermes Web UI
|
||||||
|
|
||||||
|
File:
|
||||||
|
`/etc/systemd/system/hermes-webui.service.d/resource-containment.conf`
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Service]
|
||||||
|
MemoryHigh=1G
|
||||||
|
MemoryMax=1536M
|
||||||
|
MemorySwapMax=1G
|
||||||
|
OOMPolicy=kill
|
||||||
|
```
|
||||||
|
|
||||||
|
Camofox and Web UI are allowed to restart under extreme growth instead of
|
||||||
|
consuming most of the container and taking Telegram down with them.
|
||||||
|
|
||||||
|
## I/O priority limitation
|
||||||
|
|
||||||
|
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
||||||
|
is changed and verified. CT460 exposes the I/O controller at its top cgroup,
|
||||||
|
but it is not delegated into the container's systemd service tree. Systemd
|
||||||
|
accepts the property, but no live `io.weight` control is created.
|
||||||
|
|
||||||
|
## Health check
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl is-active hermes-webui.service
|
||||||
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
||||||
|
systemctl --user is-active camofox-browser.service hermes-gateway.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Check effective limits:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl show hermes-webui.service \
|
||||||
|
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
|
||||||
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
||||||
|
systemctl --user show hermes-gateway.service \
|
||||||
|
-p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight
|
||||||
|
runuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \
|
||||||
|
systemctl --user show camofox-browser.service \
|
||||||
|
-p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected gateway values include `OOMPolicy=continue`,
|
||||||
|
`MemoryLow=536870912`, `MemoryHigh=infinity`, `MemoryMax=infinity`, and
|
||||||
|
`CPUWeight=1000`.
|
||||||
|
|
||||||
|
## Backups and rollback
|
||||||
|
|
||||||
|
Original source units are backed up at:
|
||||||
|
|
||||||
|
`/home/hermes/.hermes/backups/resource-hardening-20260725/`
|
||||||
|
|
||||||
|
To roll back, remove only these three drop-ins:
|
||||||
|
|
||||||
|
- `resource-resilience.conf` for `hermes-gateway.service`
|
||||||
|
- `resource-containment.conf` for `camofox-browser.service`
|
||||||
|
- `resource-containment.conf` for `hermes-webui.service`
|
||||||
|
|
||||||
|
Then reload the system and user systemd managers and restart only those three
|
||||||
|
services. The original source units were never modified.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/messaging-integrations]]
|
||||||
|
- [[systems/browser-backend]]
|
||||||
|
- [[runbooks/restart-hermes]]
|
||||||
|
- [[systems/tool-search]]
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
---
|
||||||
|
title: Daily Marketing Job Radar Operations and Recovery
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-27
|
||||||
|
updated: 2026-07-28
|
||||||
|
review_after: 2026-10-25
|
||||||
|
verified_on: 2026-07-28
|
||||||
|
version_applies_to: Hermes Agent cron dca8482e4f76
|
||||||
|
confidence: high
|
||||||
|
tags: [automation, cron, jobs, notion, dashboard, runbook]
|
||||||
|
sources: [live-cron-output, live-notion-api, live-dashboard-api, hermes-docs]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Daily Marketing Job Radar Operations and Recovery
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Run a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony's resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard.
|
||||||
|
|
||||||
|
## Production configuration
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|---|---|
|
||||||
|
| Cron job | `dca8482e4f76` — Daily Marketing Job Radar |
|
||||||
|
| Schedule | `0 7 * * 1-5` (07:00 AWST weekdays) |
|
||||||
|
| Delivery | Telegram group `-1004321904721`, topic `1030` |
|
||||||
|
| Skill | `job-search-automation` |
|
||||||
|
| Primary model | `openai-codex / gpt-5.6-terra` |
|
||||||
|
| First fallback | `opencode-go / mimo-v2.5` |
|
||||||
|
| Dashboard | `https://hermes.kangaroo-eel.ts.net/jobs/` |
|
||||||
|
| Dashboard backend | `127.0.0.1:9099` |
|
||||||
|
| Notion database | `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` |
|
||||||
|
|
||||||
|
Cron jobs inherit the profile-level `fallback_providers` chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks.
|
||||||
|
|
||||||
|
## Cover letters are non-negotiable
|
||||||
|
|
||||||
|
A successful radar run includes the cover-letter stage.
|
||||||
|
|
||||||
|
- Audit **every Strong Match added today**, including pages created by an interrupted earlier run.
|
||||||
|
- A valid result requires a canonical rendered `.html` URL in Notion, not plain text and not the dashboard root.
|
||||||
|
- Production template files:
|
||||||
|
- `/home/hermes/.hermes/dashboard/templates/cover-letter.html`
|
||||||
|
- `/home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx`
|
||||||
|
- The old upload-cache location is not authoritative and may be evicted.
|
||||||
|
- Required nonempty artifacts: `.txt`, `.json`, `.html`, `.docx`, `.pdf`.
|
||||||
|
- The HTML must contain zero unresolved `{{...}}` tokens and return HTTP 200.
|
||||||
|
- Retry generation once when an artifact, Notion link or HTTP check fails.
|
||||||
|
- A clean run requires `MISSING_COVER_LETTER_COUNT=0`. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted.
|
||||||
|
|
||||||
|
### Live audit — 27 July 2026
|
||||||
|
|
||||||
|
- Same-day strong matches: **1**
|
||||||
|
- Verified canonical HTML cover letters: **1**
|
||||||
|
- Missing cover letters: **0**
|
||||||
|
- The Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link.
|
||||||
|
|
||||||
|
## Required production pipeline
|
||||||
|
|
||||||
|
1. Obtain the live AWST date and calculate a seven-day freshness cutoff.
|
||||||
|
2. Search SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites.
|
||||||
|
3. Open and verify every individual advert. Search snippets are discovery evidence only.
|
||||||
|
4. Reject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages.
|
||||||
|
5. Reject **all Pacific Energy roles**. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities.
|
||||||
|
6. Accept only direct individual advert URLs:
|
||||||
|
- SEEK `/job/<numeric-id>`
|
||||||
|
- LinkedIn `/jobs/view/...<numeric-id>`
|
||||||
|
- Indeed `/viewjob?jk=<real-id>`
|
||||||
|
- direct employer career pages
|
||||||
|
- individual WA Government role pages
|
||||||
|
7. A WA Government slug page is valid without `AdvertID` in its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active **Apply Now** function.
|
||||||
|
8. Query the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations.
|
||||||
|
9. Create only verified nonduplicates and require returned Notion page IDs.
|
||||||
|
10. Refresh `POST http://127.0.0.1:9099/api/refresh`, then confirm every page and direct URL through `/api/jobs`.
|
||||||
|
11. Score newly created roles against `/home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txt` using the documented 100-point rubric. Strong Match starts at 70.
|
||||||
|
12. For each new strong match, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/home/hermes/.hermes/hermes-agent/venv/bin/python \
|
||||||
|
/home/hermes/.hermes/scripts/regenerate_cover_letter.py \
|
||||||
|
<NOTION_PAGE_ID> \
|
||||||
|
"Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template."
|
||||||
|
```
|
||||||
|
|
||||||
|
13. Verify nonempty `.txt`, `.json`, `.html`, `.docx` and `.pdf` artifacts; no unresolved `{{...}}` tokens; and a live `.html` Notion Cover Letter URL.
|
||||||
|
14. Report only facts produced by tools in that run. Never estimate counts or fabricate Notion writes.
|
||||||
|
|
||||||
|
## Dashboard behaviour
|
||||||
|
|
||||||
|
Source: `/home/hermes/.hermes/dashboard/index.html`
|
||||||
|
Backend: `/home/hermes/.hermes/scripts/job_dashboard_server.py`
|
||||||
|
|
||||||
|
### Layout (top to bottom)
|
||||||
|
1. **Hero header** — RADAR logo, name, dateline
|
||||||
|
2. **Market Intel panel** — expandable salary/location/source stats from `/api/market-intel`
|
||||||
|
3. **KPI strip** — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon
|
||||||
|
4. **Filter bar** — text search, view toggles (table/cards), Refresh button
|
||||||
|
5. **Stats strip** — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate
|
||||||
|
6. **Pipeline Funnel** — clickable per-status breakdown with bar chart
|
||||||
|
7. **Job table/cards** — sortable columns, inline status pills, cover letter links, edit actions
|
||||||
|
|
||||||
|
### Column sorting
|
||||||
|
Click any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction.
|
||||||
|
|
||||||
|
### Status management
|
||||||
|
- **Bulk update**: select jobs via checkboxes → pick status from dropdown → Apply. Sends `{ids, updates: {status}}` to `POST /api/jobs/bulk-update`.
|
||||||
|
- **Single edit**: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via `POST /api/jobs/<id>/update`.
|
||||||
|
- **Funnel/status filters**: clicking a status in the funnel or KPI strip filters the table to that status.
|
||||||
|
|
||||||
|
### Rich text notes
|
||||||
|
- Write/Preview editor in the edit modal (monospace textarea + markdown preview)
|
||||||
|
- Supports: bold `**`, italic `*`, links `[text](url)`, bullet lists `- item`
|
||||||
|
- Syncs to Notion `Notes / JD Summary` rich_text property on every save
|
||||||
|
|
||||||
|
### Document attachments
|
||||||
|
- Attach any URL (documents, links, files) to a job entry
|
||||||
|
- Stored as **bookmark blocks** on the Notion page body (not a property)
|
||||||
|
- Lazy-loaded when edit modal opens: `GET /api/jobs/<id>/attachments`
|
||||||
|
- Add: `POST /api/jobs/<id>/attachments` with `{name, url}`
|
||||||
|
- Delete: `POST /api/jobs/<id>/attachments/delete` with `{block_id}`
|
||||||
|
- Visible in Notion when you open the page
|
||||||
|
|
||||||
|
### Cover letter integration
|
||||||
|
- Regenerate button (🔄) on each row calls `POST /api/jobs/<id>/cover-letter/regenerate`
|
||||||
|
- CL URL stored in Notion `Cover Letter` property and rendered as clickable link in table
|
||||||
|
- All 5 artifact formats: TXT, JSON, HTML, DOCX, PDF
|
||||||
|
|
||||||
|
### Market Intel
|
||||||
|
- Expandable panel at top of page
|
||||||
|
- Data from `GET /api/market-intel`: total tracked, salary stats (avg/median/min/max), counts
|
||||||
|
- Lazy-loaded on expand; also refreshes on ↺ Refresh
|
||||||
|
|
||||||
|
### API endpoints
|
||||||
|
| Endpoint | Method | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `/api/jobs` | GET | All jobs (cached, paginated from Notion) |
|
||||||
|
| `/api/stats` | GET | Aggregate statistics |
|
||||||
|
| `/api/refresh` | POST | Force Notion cache refresh |
|
||||||
|
| `/api/market-intel` | GET | Salary/location/source analytics |
|
||||||
|
| `/api/jobs/<id>/update` | POST | Update single job properties |
|
||||||
|
| `/api/jobs/<id>/attachments` | GET | Fetch page block attachments |
|
||||||
|
| `/api/jobs/<id>/attachments` | POST | Add bookmark attachment to page |
|
||||||
|
| `/api/jobs/<id>/attachments/delete` | POST | Remove block from page |
|
||||||
|
| `/api/jobs/<id>/cover-letter/regenerate` | POST | Regenerate cover letter |
|
||||||
|
| `/api/jobs/<id>/match` | GET | Match analysis for a job |
|
||||||
|
| `/api/jobs/bulk-update` | POST | Update multiple jobs at once |
|
||||||
|
| `/api/jobs/create` | POST | Create new job in Notion |
|
||||||
|
|
||||||
|
- Company names and job titles with a valid `URL` render as blue external links and open the direct advert in a new tab.
|
||||||
|
- URL values are protocol-validated; only `http:` and `https:` are rendered as links.
|
||||||
|
- KPI/stat cards are one-click filters with an active highlight:
|
||||||
|
- Total Jobs
|
||||||
|
- Backlog
|
||||||
|
- Applied
|
||||||
|
- Interviews
|
||||||
|
- Offers
|
||||||
|
- Closing Soon
|
||||||
|
- This Week
|
||||||
|
- Stale
|
||||||
|
- Strong Matches
|
||||||
|
- Past Closing
|
||||||
|
- Application Rate
|
||||||
|
- Funnel status rows are also one-click exact-status filters.
|
||||||
|
- Existing status editing remains through the job edit modal and bulk-action controls, which write back to Notion through `/api/update`.
|
||||||
|
- The frontend filter/link test harness is `/tmp/radar_dashboard_test.js` (temporary; recreate if absent).
|
||||||
|
|
||||||
|
## 27 July 2026 incident
|
||||||
|
|
||||||
|
### Symptoms
|
||||||
|
|
||||||
|
- The morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony's redundancy.
|
||||||
|
- Aggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts.
|
||||||
|
- Counts and quality signals were presented without a demonstrated live Notion query/write trail.
|
||||||
|
- The dashboard had valid URLs for almost every record, but the frontend never rendered `job.url`; users could not open the actual adverts.
|
||||||
|
- The earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage.
|
||||||
|
|
||||||
|
### Recovery
|
||||||
|
|
||||||
|
- Archived all **16** records created by the contaminated morning run: 16 successful, zero failures.
|
||||||
|
- Replaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates.
|
||||||
|
- Pinned the cron primary to `gpt-5.6-terra`.
|
||||||
|
- Added and directly probed `opencode-go / mimo-v2.5`; it returned `MIMO_FALLBACK_OK`, then was inserted first in the inherited fallback chain.
|
||||||
|
- Repaired `regenerate_cover_letter.py`:
|
||||||
|
- absolute Hermes paths rather than `~` expansion
|
||||||
|
- one correct command dispatcher rather than two conflicting `__main__` blocks
|
||||||
|
- Repaired dashboard direct links and one-click quick filters.
|
||||||
|
- The corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats.
|
||||||
|
- A follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added.
|
||||||
|
|
||||||
|
### Final verification
|
||||||
|
|
||||||
|
- Dashboard API: 166 records
|
||||||
|
- Records created on 27 July after cleanup: 4
|
||||||
|
- Pacific Energy records created on 27 July: 0
|
||||||
|
- Records with direct URLs: 164 of 166
|
||||||
|
- Strong-match cover letter artifact sizes were nonzero for all five formats
|
||||||
|
- Unresolved template tokens: 0
|
||||||
|
- Dashboard, API and generated cover-letter URLs returned HTTP 200
|
||||||
|
- Synthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks
|
||||||
|
|
||||||
|
## Tailscale route recovery found during verification
|
||||||
|
|
||||||
|
The dashboard's local service was healthy while the tailnet URL failed because the CT460 `:443` Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public `:8443` Funnel route.
|
||||||
|
|
||||||
|
`/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` now atomically restores both:
|
||||||
|
|
||||||
|
- tailnet-only `:443`: `/`, `/jobs`, `/desktop`, `/vnc-camofox`, `/evening-grid.html`
|
||||||
|
- public `:8443`: `/webhook` → `http://127.0.0.1:8085/webhook`
|
||||||
|
|
||||||
|
Important: `tailscale serve reset` clears Funnel state too. Never restore only the 443 routes and assume 8443 survived.
|
||||||
|
|
||||||
|
## Verification commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Cron configuration
|
||||||
|
hermes cron list
|
||||||
|
|
||||||
|
# Fallback order
|
||||||
|
hermes fallback list
|
||||||
|
|
||||||
|
# Dashboard/API
|
||||||
|
curl -I https://hermes.kangaroo-eel.ts.net/jobs/
|
||||||
|
curl -sS https://hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length'
|
||||||
|
|
||||||
|
# Local service
|
||||||
|
ss -ltnp 'sport = :9099'
|
||||||
|
curl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length'
|
||||||
|
|
||||||
|
# Route table
|
||||||
|
tailscale serve status
|
||||||
|
|
||||||
|
# Atomic route rebuild
|
||||||
|
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
|
||||||
|
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backups from the incident
|
||||||
|
|
||||||
|
- `/home/hermes/workspace/index.html.radar-links-backup-20260727`
|
||||||
|
- `/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727`
|
||||||
|
- `/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727`
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[systems/scheduled-tasks]]
|
||||||
|
- [[infrastructure/tailscale]]
|
||||||
|
- [[systems/model-providers]]
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
---
|
||||||
|
title: Multiplexer Setup (Multi-Profile Gateway)
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-28
|
||||||
|
updated: 2026-07-28
|
||||||
|
verified_on: 2026-07-28
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Multiplexer Setup
|
||||||
|
|
||||||
|
## What It Is
|
||||||
|
|
||||||
|
One gateway process serving multiple Hermes profiles through a single Telegram bot. Messages are routed to the correct profile by `profile_routes` (chat_id → profile).
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Telegram Bot (single token)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Default Gateway (multiplex_profiles: true)
|
||||||
|
│
|
||||||
|
├── profile_routes: chat_id → profile
|
||||||
|
│
|
||||||
|
├── default profile (Rhino chat: -1004321904721)
|
||||||
|
└── ops profile (Tusk ops: -1003914987043, -1003932503629)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Config Locations
|
||||||
|
|
||||||
|
### Default profile: `~/.hermes/config.yaml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
gateway:
|
||||||
|
multiplex_profiles: true
|
||||||
|
profile_routes:
|
||||||
|
- name: ops-group
|
||||||
|
platform: telegram
|
||||||
|
chat_id: '-1003914987043'
|
||||||
|
profile: ops
|
||||||
|
- name: ops-group-2
|
||||||
|
platform: telegram
|
||||||
|
chat_id: '-1003932503629'
|
||||||
|
profile: ops
|
||||||
|
|
||||||
|
telegram:
|
||||||
|
allowed_chats: '-1004321904721,-1003914987043,-1003932503629'
|
||||||
|
group_allowed_chats: '-1004321904721,-1003914987043,-1003932503629'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ops profile: `~/.hermes/profiles/ops/config.yaml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
telegram:
|
||||||
|
allowed_chats:
|
||||||
|
- '-1003914987043'
|
||||||
|
- '-1003932503629'
|
||||||
|
group_allowed_chats:
|
||||||
|
- '-1003914987043'
|
||||||
|
- '-1003932503629'
|
||||||
|
|
||||||
|
platforms:
|
||||||
|
telegram:
|
||||||
|
enabled: false # ← DISABLED — multiplexer handles this
|
||||||
|
```
|
||||||
|
|
||||||
|
## The Silent-Drop Pitfall
|
||||||
|
|
||||||
|
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
||||||
|
|
||||||
|
**Moral:** When adding a new chat for a secondary profile, add it to BOTH profiles' `allowed_chats`.
|
||||||
|
|
||||||
|
## Adding a New Group
|
||||||
|
|
||||||
|
1. Add the chat_id to the default profile's `profile_routes`
|
||||||
|
2. Add the chat_id to the ops profile's `allowed_chats` and `group_allowed_chats`
|
||||||
|
3. **Add the chat_id to the default profile's `allowed_chats` and `group_allowed_chats`** (comma-separated list)
|
||||||
|
4. Restart the gateway: `systemctl --user restart hermes-gateway`
|
||||||
|
|
||||||
|
## Restarting
|
||||||
|
|
||||||
|
From an SSH shell (outside the agent process):
|
||||||
|
```bash
|
||||||
|
systemctl --user restart hermes-gateway
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[hermes-agent]]
|
||||||
|
- [[messaging-integrations]]
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
---
|
||||||
|
title: Provider Health Check
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, providers, health, routing]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Provider Health Check
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Verify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Models list is slow or blank in Hermes
|
||||||
|
- Fallback model retries happen immediately
|
||||||
|
- Chat returns 5xx or gateway errors
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Network access from this host to `litellm:4000`, `omniroute:20128`, `opencode.ai`, `api.telegram.org`, Telegram fallback IPs
|
||||||
|
- Hermes config slocated at `~/.hermes/config.yaml`
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Check local Hermes services
|
||||||
|
```bash
|
||||||
|
hermes gateway status
|
||||||
|
hermes dashboard status
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Check main provider reachability
|
||||||
|
```bash
|
||||||
|
python3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"
|
||||||
|
```
|
||||||
|
Current install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.
|
||||||
|
|
||||||
|
### 3. Check OmniRoute reachability
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
import urllib.request, json, yaml
|
||||||
|
with open('/home/hermes/.hermes/config.yaml') as f:
|
||||||
|
cfg = yaml.safe_load(f)
|
||||||
|
key = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]
|
||||||
|
req = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})
|
||||||
|
with urllib.request.urlopen(req, timeout=5) as r:
|
||||||
|
data = json.loads(r.read())
|
||||||
|
print('OmniRoute OK, models:', len(data.get('data', [])))
|
||||||
|
"
|
||||||
|
```
|
||||||
|
Current verified result: reachable and returns a non-empty model catalog.
|
||||||
|
|
||||||
|
### 4. Check fallback provider endpoints
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://opencode.ai/zen/v1/models
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://inference-api.nousresearch.com/v1/models
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" --max-time 5 https://ollama.com/v1/models
|
||||||
|
```
|
||||||
|
Any non-200 here means that fallback path is currently offline.
|
||||||
|
|
||||||
|
### 5. Check WebUI health
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
|
||||||
|
```
|
||||||
|
Expected: `200`
|
||||||
|
|
||||||
|
### 6. Check gateway platform reachability
|
||||||
|
Tail the gateway journal for errors:
|
||||||
|
```bash
|
||||||
|
journalctl --user -u hermes-gateway.service -n 100 --no-pager
|
||||||
|
```
|
||||||
|
Look for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- Gateway status shows `active (running)`.
|
||||||
|
- LiteLLM 401 from unauthenticated check is expected.
|
||||||
|
- OmniRoute returns model count > 0.
|
||||||
|
- Fallback URLs return 200 or expected auth codes.
|
||||||
|
- WebUI `/health` returns `200`.
|
||||||
|
- Gateway journal shows no red alarm pattern.
|
||||||
|
|
||||||
|
## Current install references
|
||||||
|
- Hermes: v0.19.0, git install: `/home/hermes/.hermes/hermes-agent`
|
||||||
|
- LiteLLM base URL: `http://litellm:4000/v1`
|
||||||
|
- OmniRoute base URL: `http://omniroute:20128/v1`
|
||||||
|
- OmniRoute direct IP: `100.88.81.19:20128`
|
||||||
|
- WebUI: `http://0.0.0.0:8787/health`
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- This read-only runbook has no rollback side effects
|
||||||
|
- If you change fallback providers during remediation, record and revert via `hermes fallback list` + previous wiki state
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[model-providers]]
|
||||||
|
- [[messaging-integrations]]
|
||||||
|
- [[restart-hermes]]
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
---
|
||||||
|
title: Recover a Docker service
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
tags: [runbook, docker, recovery]
|
||||||
|
sources: []
|
||||||
|
confidence: high
|
||||||
|
---
|
||||||
|
# Recover Docker service
|
||||||
|
## Procedure
|
||||||
|
1. Identify the container.
|
||||||
|
2. Inspect status and last 100 log lines.
|
||||||
|
3. If unhealthy, restart then re-check.
|
||||||
|
4. If still failing, inspect image and mounts.
|
||||||
|
## Verification
|
||||||
|
Container shows healthy after recovery probe reboots.
|
||||||
|
## Last successfully used
|
||||||
|
2026-07-22
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
---
|
||||||
|
title: Restart Browser Helper
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, browser, cua-driver, camofox]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Restart Browser Helper
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Restart the computer-use browser stack that Hermes uses for web automation, screenshots, and VNC/noVNC access.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Browser screenshots are failing
|
||||||
|
- `cua-driver` MCP tools return errors
|
||||||
|
- VNC/noVNC is unreachable on `5901` / `6081`
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Display helper already running: `Xvfb :99`, x11vnc on `127.0.0.1:5901`, noVNC at `127.0.0.1:6081`
|
||||||
|
- `cua-driver` binary installed at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
|
||||||
|
- Browser VM/desktop helper is expected but not mandatory
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Inspect current browser helpers
|
||||||
|
```bash
|
||||||
|
ps aux | grep -E "cua-driver serve|x11vnc|fluxbox|vnc-watcher" | grep -v grep
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Stop cua-driver only
|
||||||
|
```bash
|
||||||
|
pkill -f "cua-driver serve"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Start cua-driver
|
||||||
|
```bash
|
||||||
|
/home/hermes/.local/bin/cua-driver serve
|
||||||
|
```
|
||||||
|
Run in background or a tmux/screen session if needed.
|
||||||
|
|
||||||
|
### 4. Verify ports remain available
|
||||||
|
```bash
|
||||||
|
nc -z 127.0.0.1 5901 && echo "x11vnc ok" || echo "x11vnc missing"
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:6081/
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Restart full browser VM stack only if step 4 fails
|
||||||
|
```bash
|
||||||
|
pkill -f "Xvfb :99"
|
||||||
|
pkill -f "x11vnc"
|
||||||
|
pkill -f "fluxbox"
|
||||||
|
/home/hermes/.local/bin/cua-driver serve
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `cua-driver serve` process is present in `ps`
|
||||||
|
- `5901` and `6081` are reachable
|
||||||
|
- No zombie accumulation in browser/desktop processes
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- If browser automation is not mission-critical, skip full VM restart and continue without it
|
||||||
|
- Do not stop gateway or dashboard unless necessary
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- This runbook assumes the standalone desktop helper is running separately from Hermes core
|
||||||
|
- The browser backend path can work without browser automation; this stack is optional depending on task
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[browser-backend]]
|
||||||
|
- [[restart-hermes]]
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
---
|
||||||
|
title: Restart Hermes
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, hermes, restart]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Restart Hermes
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Restart the Hermes agent, gateway, dashboard, and browser helper processes cleanly.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Hermes stops responding in chat or WebUI
|
||||||
|
- Tools fail mid-session but the agent is otherwise running
|
||||||
|
- After an update or config change
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- User account `hermes` with access to systemd --user
|
||||||
|
- No active backups in progress
|
||||||
|
- If using a browser session, warn the user before restarting dashboards
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Identify the active Hermes processes
|
||||||
|
```bash
|
||||||
|
ps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep
|
||||||
|
```
|
||||||
|
Expected active pieces on this install:
|
||||||
|
- gateway: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`
|
||||||
|
- dashboard: `/home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...`
|
||||||
|
- webui: `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py`
|
||||||
|
- camofox/cua-driver: `/home/hermes/.local/bin/cua-driver serve`
|
||||||
|
- x11vnc: `-display :99 ... 5901`
|
||||||
|
|
||||||
|
### 2. Restart the gateway
|
||||||
|
```bash
|
||||||
|
hermes gateway restart
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Restart the dashboard if needed
|
||||||
|
```bash
|
||||||
|
hermes dashboard restart
|
||||||
|
```
|
||||||
|
If that fails, stop and rerun:
|
||||||
|
```bash
|
||||||
|
hermes dashboard stop
|
||||||
|
hermes dashboard --host 0.0.0.0 --port 9119 --no-open
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Restart the browser helper if needed
|
||||||
|
```bash
|
||||||
|
pkill -f "cua-driver serve"
|
||||||
|
/home/hermes/.local/bin/cua-driver serve
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Restart the WebUI if needed
|
||||||
|
Find its PID, then restart standalone:
|
||||||
|
```bash
|
||||||
|
pkill -f "hermes-webui/server.py"
|
||||||
|
/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
```bash
|
||||||
|
hermes gateway status
|
||||||
|
hermes dashboard status
|
||||||
|
ss -tlnp | grep -E "8787|9119|5901|6081"
|
||||||
|
ps aux | grep -E "gateway run|dashboard|cua-driver serve|server.py" | grep -v grep
|
||||||
|
```
|
||||||
|
Current known-good ports after verified restart:
|
||||||
|
- gateway: active via `hermes-gateway.service`
|
||||||
|
- dashboard: `0.0.0.0:9119`
|
||||||
|
- webui: `0.0.0.0:8787`
|
||||||
|
- VNC: `5901`
|
||||||
|
- noVNC: `6081`
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- If restart fails, review recent config changes and consider restoring `config.yaml.bak.*`
|
||||||
|
- Do not delete old config files without explicit confirmation
|
||||||
|
- Re-run backup first if you plan to rollback config
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- Last tested: 2026-07-22 on debian 13, user-mode systemd
|
||||||
|
- WebUI logs: `/home/hermes/.hermes/webui/bootstrap-8787.log`
|
||||||
|
- Defunct `<defunct>` processes in process list are expected zombie shells from prior runs and do not require action unless they pile up
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[runbooks/backup-wiki]]
|
||||||
|
- [[runbooks/update-hermes-safely]]
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
---
|
||||||
|
title: Restore OpenRouter connectivity
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
tags: [runbook, openrouter, recovery]
|
||||||
|
sources: []
|
||||||
|
confidence: high
|
||||||
|
---
|
||||||
|
# Restore OpenRouter connectivity
|
||||||
|
## Symptoms
|
||||||
|
- TLS connection hangs
|
||||||
|
- Model requests time out
|
||||||
|
- Other providers still work
|
||||||
|
## Procedure
|
||||||
|
1. Check session activity; ensure previous HTTPS calls completed.
|
||||||
|
2. Review last WARP errors.
|
||||||
|
3. Restart the Hermes process with the GoOp tool.
|
||||||
|
4. Send a small test completion request.
|
||||||
|
5. Confirm tool calling works.
|
||||||
|
## Verification
|
||||||
|
A test prompt returns successfully through OpenRouter.
|
||||||
|
## Escalation
|
||||||
|
If the failure persists, check DNS, certificate expiry and provider status.
|
||||||
|
## Last successfully used
|
||||||
|
2026-06-22
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
title: Test OpenRouter
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, openrouter, providers, fallback]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Test OpenRouter
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Test whether OpenRouter is available to Hermes and whether fallback providers are actually callable in this installation.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- Hermes reports fallback model errors
|
||||||
|
- `hermes fallback list` shows expected providers, but actual requests fail
|
||||||
|
- User wants to confirm OpenRouter is usable without relying on catalog names
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Hermes CLI available: `/home/hermes/.local/bin/hermes`
|
||||||
|
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. List fallback providers
|
||||||
|
```bash
|
||||||
|
hermes fallback list
|
||||||
|
```
|
||||||
|
Expected active fallbacks on this install:
|
||||||
|
1. `deepseek-v4-flash-free` via opencode-zen
|
||||||
|
2. `stepfun/step-3.7-flash:free` via nous
|
||||||
|
3. `qwen3.5:397b` via ollama-cloud
|
||||||
|
4. `poolside/laguna-m.1:free` via kilocode
|
||||||
|
5. `or-inclusionai_ling-2_6-1t` via custom/litellm
|
||||||
|
|
||||||
|
### 2. Test OpenRouter-style direct access if present
|
||||||
|
```bash
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" --max-time 8 https://openrouter.ai/api/v1/models
|
||||||
|
```
|
||||||
|
Current install state as of last run: OpenRouter is not configured as a standalone provider in Hermes, so this is an optional confirmation step only.
|
||||||
|
|
||||||
|
### 3. Test each verified fallback URL
|
||||||
|
```bash
|
||||||
|
for url in \
|
||||||
|
"https://opencode.ai/zen/v1/models" \
|
||||||
|
"https://inference-api.nousresearch.com/v1/models" \
|
||||||
|
"https://ollama.com/v1/models" \
|
||||||
|
"https://api.kilo.ai/api/gateway/models"; do
|
||||||
|
printf "%s -> " "$url"
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"
|
||||||
|
echo
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Check Hermes fallback config reflects reality
|
||||||
|
```bash
|
||||||
|
grep -A5 "^fallback:" ~/.hermes/config.yaml
|
||||||
|
```
|
||||||
|
Verify provider names and URLs in config match what the user wants to use.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `hermes fallback list` shows the exact same providers as the live config
|
||||||
|
- Each URL returns an HTTP response within timeout
|
||||||
|
- No leftover undefined OpenRouter provider entries exist in `custom_providers`
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- No system changes in this runbook
|
||||||
|
- If a provider looks wrong but is part of live chat behavior, treat it like a config issue, not a transient health failure
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- In this installation, OpenRouter is not a live first-class provider in `custom_providers`
|
||||||
|
- This runbook tests confirmed fallback paths, not assumed ones
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[provider-health]]
|
||||||
|
- [[model-providers]]
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
title: Troubleshoot Tool Loop
|
||||||
|
type: runbook
|
||||||
|
status: unresolved
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
review_after: 2026-08-22
|
||||||
|
confidence: low
|
||||||
|
tags: [runbook, hermes]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
||||||
|
|
||||||
|
# Troubleshoot Tool Loop
|
||||||
|
## Symptoms
|
||||||
|
Agent loops on one tool or endpoint.
|
||||||
|
## Procedure
|
||||||
|
1. Inspect recent Hermes agent logs.
|
||||||
|
2. Identify the looping tool path.
|
||||||
|
3. Restart worker or flush stuck request.
|
||||||
|
4. Re-run limited test scenario.
|
||||||
|
## Verification
|
||||||
|
Agent handles the same scenario without repeating.
|
||||||
|
## Last successfully used
|
||||||
|
2026-07-22
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
---
|
||||||
|
title: Update Hermes safely
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
last_tested: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [runbook, hermes, update]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Update Hermes safely
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Update Hermes to a newer release without losing configuration, skills, or sessions.
|
||||||
|
|
||||||
|
## Symptoms that match this runbook
|
||||||
|
- User wants to move to a newer Hermes version
|
||||||
|
- `hermes version` shows an older build than expected
|
||||||
|
- Update-related errors after manual changes in the install directory
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
- Hermes install method: git-based install at `/home/hermes/.hermes/hermes-agent`
|
||||||
|
- Working Python virtualenv at `/home/hermes/.hermes/hermes-agent/venv`
|
||||||
|
- Available disk space in Hermes install and backup paths
|
||||||
|
- No active Hermes chat/webui shell issues before starting
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Create a backup first
|
||||||
|
```bash
|
||||||
|
hermes backup --quick --label pre-update-$(date +%Y%m%d)
|
||||||
|
```
|
||||||
|
If you need a full backup instead:
|
||||||
|
```bash
|
||||||
|
hermes backup --label pre-update-$(date +%Y%m%d)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Check whether an update is available
|
||||||
|
```bash
|
||||||
|
hermes update --check
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Review configuration version and backups
|
||||||
|
```bash
|
||||||
|
python3 -c "import yaml; print(yaml.safe_load(open('/home/hermes/.hermes/config.yaml')).get('_config_version'))"
|
||||||
|
ls -la ~/.hermes/config.yaml.bak*
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Run the update with noninteractive flags
|
||||||
|
```bash
|
||||||
|
cd /home/hermes/.hermes/hermes-agent
|
||||||
|
git status --short
|
||||||
|
hermes update --yes --branch main
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Verify the new version
|
||||||
|
```bash
|
||||||
|
hermes version
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Restart the required Hermes pieces
|
||||||
|
```bash
|
||||||
|
hermes gateway status
|
||||||
|
hermes dashboard status
|
||||||
|
hermes gateway restart
|
||||||
|
hermes dashboard restart
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. Sanity check health
|
||||||
|
```bash
|
||||||
|
hermes gateway status
|
||||||
|
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- `hermes version` shows a newer version than before
|
||||||
|
- `hermes gateway status` is active
|
||||||
|
- WebUI `/health` returns `200`
|
||||||
|
- Providers reachable with [[provider-health]]
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
- Rollback Hermes config:
|
||||||
|
```bash
|
||||||
|
ls -lt ~/.hermes/config.yaml.bak*
|
||||||
|
cp ~/.hermes/config.yaml.bak.<timestamp> ~/.hermes/config.yaml
|
||||||
|
hermes gateway restart
|
||||||
|
hermes dashboard restart
|
||||||
|
```
|
||||||
|
- Rollback entire Hermes install requires git revert or reinstall from previous checkout
|
||||||
|
- If update warns about config migration, revert the backup and re-run manually
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- Hermes version on this install: v0.19.0 (2026.7.20)
|
||||||
|
- Install directory: `/home/hermes/.hermes/hermes-agent`
|
||||||
|
- Update preserves venv by default unless `--force-venv` is used
|
||||||
|
|
||||||
|
## Last tested
|
||||||
|
2026-07-22 on debian 13, git-based Hermes install
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[restart-hermes]]
|
||||||
|
- [[backup-wiki]]
|
||||||
|
- [[provider-health]]
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
---
|
||||||
|
title: Browser Backend
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [browser, camofox, automation]
|
||||||
|
sources: [live-probe, config, session-history]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Browser Backend
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Browser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.
|
||||||
|
|
||||||
|
## Current Configuration (as of 2026-08-02)
|
||||||
|
- **Primary backend:** Camoufox (Firefox-based, anti-fingerprint) on CT450
|
||||||
|
- **Provider config:** `browser.cloud_provider: camofox`, `camera_provider: camofox`
|
||||||
|
- **REST API:** `localhost:9093` (profile `anthony`, sessionKey `anthony`)
|
||||||
|
- **CDP WebSocket:** `localhost:9377` (via Camoufox's Playwright bridge)
|
||||||
|
- **VNC viewer:** `https://ct460.kangaroo-eel.ts.net/vnc-camofox` (port 6080)
|
||||||
|
- **Steel Browser (CT201):** Still running but Hermes not connected — plugin disabled
|
||||||
|
|
||||||
|
## Camoufox (CT450)
|
||||||
|
- **Host:** CT450 (same host as Hermes — no network hop)
|
||||||
|
- **Deployment:** Systemd service `camofox-browser.service`
|
||||||
|
- **Engine:** Camoufox (custom Firefox build with anti-detection fingerprinting)
|
||||||
|
- **Display:** Headed via Xvfb `:99` (1600×1200×24) + fluxbox window manager
|
||||||
|
- **Auth:** Persistent profile `anthony` — authenticated into Uber Eats, LinkedIn, etc.
|
||||||
|
- **Connection:** REST API at `localhost:9093` for session management, CDP WebSocket at `localhost:9377` for browser tools
|
||||||
|
- **Last verified:** Jul 30 — `browserConnected: true`, `sessions: 1`, `tabs: 1`, active 200s on scroll/navigate
|
||||||
|
|
||||||
|
### Access
|
||||||
|
| Endpoint | URL | Used by |
|
||||||
|
|---|---|---|
|
||||||
|
| REST API | `http://localhost:9093` | Hermes browser tools (navigate, click, snapshot) |
|
||||||
|
| CDP WebSocket | `ws://localhost:9377` | Direct CDP control |
|
||||||
|
| VNC | `https://ct460.kangaroo-eel.ts.net/vnc-camofox` | Visual session viewer |
|
||||||
|
| Camoufox UI | `http://localhost:9378` | Internal management |
|
||||||
|
|
||||||
|
### Config
|
||||||
|
- `cloud_provider: camofox` in `~/.hermes/config.yaml`
|
||||||
|
- `CAMOFOX_URL=http://localhost:9093` in `~/.hermes/.env`
|
||||||
|
- `CAMOFOX_API_KEY` in `~/.hermes/.env`
|
||||||
|
- Profile: `anthony` (502913bf) — NOT `persist-anthony`
|
||||||
|
|
||||||
|
## Steel Browser (CT201) — Disabled
|
||||||
|
- **Status:** Still running on CT201, Hermes not connected since 2026-07-30
|
||||||
|
- **Plugin:** `browser-steel` at `/home/hermes/.hermes/plugins/browser-steel/` — disabled
|
||||||
|
- **CDP:** `http://100.96.244.39:9223`
|
||||||
|
- **REST API:** `http://100.96.244.39:3000`
|
||||||
|
- **UI:** `https://runtipi.kangaroo-eel.ts.net/steel`
|
||||||
|
- **See:** [[comparisons/steel-vs-camoufox]] for the full experiment story
|
||||||
|
|
||||||
|
## History
|
||||||
|
- **2026-07-22:** Camoufox primary, cua-driver bridge
|
||||||
|
- **2026-07-29:** Steel Browser deployed on CT201, became primary
|
||||||
|
- **2026-07-30:** Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.
|
||||||
|
- **2026-08-02:** Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[comparisons/steel-vs-camoufox]]
|
||||||
|
- [[concepts/browser-automation]]
|
||||||
|
- [[decisions/rollback-steel-to-camofox]]
|
||||||
|
- [[runbooks/restart-browser]]
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
---
|
||||||
|
title: Cloudflare Tunnel
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [system, network, cloudflare]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Cloudflare Tunnel
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Exposes internal homelab services, including Hermes, to the public internet without opening inbound firewall ports.
|
||||||
|
|
||||||
|
## Location
|
||||||
|
- **Runs on:** ct207 "webserver" (Tailscale `100.82.40.23`), **not** on this host.
|
||||||
|
- **Binary:** `/bin/cloudflared`
|
||||||
|
- **Service:** `cloudflared.service` (systemd, active/running on ct207)
|
||||||
|
- **Tunnel name / ID:** `ct207-tunnel` / `339a6757-5574-4723-a16c-da4769535caa`
|
||||||
|
- **Config:** `/etc/cloudflared/config.yml` on ct207; credentials file `339a6757-5574-4723-a16c-da4769535caa.json` in the same directory (not read/copied here — credential material, not a fact to mirror).
|
||||||
|
|
||||||
|
## Current configuration
|
||||||
|
- `originRequest`: `noTLSVerify: true`, `disableChunkedEncoding: true`, `http2Origin: false` (global defaults; some hostnames override per-entry)
|
||||||
|
- Ingress is a flat list of `hostname → service` mappings ending in `service: http_status:404` catch-all — see [[infrastructure/domains-and-tunnels]] for the full list and the entries specific to Hermes.
|
||||||
|
- Two Hermes-relevant entries point at this host's Tailscale IP (`100.118.5.51`): `hermes.martinwa.org` (port 19119) and `hermes-desktop.martinwa.org` (port 5174).
|
||||||
|
|
||||||
|
## Connected systems
|
||||||
|
- [[systems/hermes-agent]] — the service actually reached through this tunnel
|
||||||
|
- [[infrastructure/domains-and-tunnels]] — full hostname table
|
||||||
|
- [[infrastructure/network-map]]
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- This host cannot restart or inspect the tunnel directly (no `cloudflared` installed here); a tunnel-side problem requires access to ct207.
|
||||||
|
- `mcp-portal.martinwa.org`, referenced in Hermes's own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — **unresolved**, may be on a different tunnel/host.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/diagnose-docker]] (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents `systemctl restart cloudflared` on ct207, since Hermes has no direct access to perform it)
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
---
|
||||||
|
title: Current Hermes Profile
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [system, hermes, profile]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Current Hermes Profile
|
||||||
|
|
||||||
|
## Active Profile
|
||||||
|
- **Name:** `default`
|
||||||
|
- **Location:** `~/.hermes/profiles/default/`
|
||||||
|
- **No other profiles confirmed**
|
||||||
|
|
||||||
|
## Key Settings
|
||||||
|
- **Model default:** `freellmapi-auto`
|
||||||
|
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
|
||||||
|
- **Switch persistence:** `persist_switch_by_default: false`
|
||||||
|
- **Timezone:** `Australia/Perth`
|
||||||
|
- **Approvals mode:** off
|
||||||
|
- **Memory:** Mnemosyne, session-scoped, auto-sleep enabled
|
||||||
|
- **Pet slug:** none
|
||||||
|
- **Skin:** slate
|
||||||
|
|
||||||
|
## Paths
|
||||||
|
- Config: `~/.hermes/config.yaml`
|
||||||
|
- Working directory: `/home/hermes/workspace`
|
||||||
|
- Install: `~/.hermes/hermes-agent/`
|
||||||
|
- Bin: `~/.local/bin/hermes`
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[model-providers]]
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
---
|
||||||
|
title: Guanaco
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-08-14
|
||||||
|
updated: 2026-08-14
|
||||||
|
verified_on: 2026-08-14
|
||||||
|
confidence: high
|
||||||
|
tags: [llm, proxy, router, guanaco, ollama]
|
||||||
|
sources: [live-probe, session-history]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Guanaco
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
LLM proxy/router providing model routing and tool-call formatting for the Ollama-based inference stack on CT205.
|
||||||
|
|
||||||
|
## Current Configuration
|
||||||
|
- **Host:** CT205 (`ollama` LXC container) on Proxmox
|
||||||
|
- **Version:** v0.8.14 (updated from v0.8.11 on 2026-08-14)
|
||||||
|
- **Repo path (on CT205):** `/root/.guanaco/repo`
|
||||||
|
- **API endpoint:** `ollama:11434` (DNS) / `localhost:11434`
|
||||||
|
- **Key component:** `cmdcode_client.py` — Guanaco's tool-call formatting layer (DSML)
|
||||||
|
|
||||||
|
## Version History
|
||||||
|
|
||||||
|
### v0.8.14 (current, updated 2026-08-14)
|
||||||
|
- **DSML tag leakage fixes** (cumulative across v0.8.12–v0.8.14):
|
||||||
|
- v0.8.12 — incomplete DSML blocks leaking raw tags as content
|
||||||
|
- v0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo `invoke`
|
||||||
|
- v0.8.14 — `<dsml_ignore>` blocks, bare parameter values, stray `>` cleanup
|
||||||
|
- **Dashboard performance fix** (`analytics.py`): fixed wrong column names in optimized history query, cached content size, stopped `SELECT *` on history table
|
||||||
|
- No new features, no breaking config changes — low-risk incremental bug-fix release
|
||||||
|
- Local `app.py` modification (working-tree change) survived the tag checkout intact
|
||||||
|
|
||||||
|
### v0.8.11 (previous)
|
||||||
|
- Had known DSML tag leakage issues where malformed tool-call output could leak raw `<dsml_...>` syntax into model context stream
|
||||||
|
|
||||||
|
## What DSML Is
|
||||||
|
DSML = "Daily Smart Messaging Log" — Guanaco's tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (`<dsml_invoke>`, bare `>` characters) into the conversation context, which confused downstream models.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[infrastructure/proxmox]] — CT205 in LXC container table
|
||||||
|
- [[systems/ollama]] — Ollama inference stack (if exists)
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
---
|
||||||
|
title: Headless Server
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [system, browser]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Headless Server
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Headless X11 environment backing Hermes's browser automation and any GUI tooling.
|
||||||
|
|
||||||
|
## Location
|
||||||
|
- **Host:** this container (ct460, `hermes`), no separate VM/LXC — headless stack runs directly alongside the agent.
|
||||||
|
- **Display:** `Xvfb :99`, resolution `1600x1200x24`, started with `-ac -nolisten tcp`
|
||||||
|
|
||||||
|
## Current configuration (verified via `ps aux`, 2026-07-22)
|
||||||
|
| Process | Detail |
|
||||||
|
|---|---|
|
||||||
|
| `Xvfb` | `:99 -screen 0 1600x1200x24 -ac -nolisten tcp` |
|
||||||
|
| `fluxbox` | window manager on `:99` |
|
||||||
|
| `x11vnc` | `-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw` — **no VNC password set** |
|
||||||
|
| `websockify` ×2 | `127.0.0.1:6080 → 127.0.0.1:5900` and `127.0.0.1:6081 → 127.0.0.1:5901`, serving noVNC (`/usr/share/novnc`) |
|
||||||
|
|
||||||
|
Two VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (`:99`→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn't independently re-verified this pass — **unresolved** whether it's a second live display or a leftover process.
|
||||||
|
|
||||||
|
## Connected systems
|
||||||
|
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
|
||||||
|
- [[infrastructure/hosts]]
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- `x11vnc -nopw`: VNC is unauthenticated. It's bound to all interfaces (`0.0.0.0:5901` and `[::]:5901`), not just loopback — reachable from the LAN and, if any tunnel/firewall rule forwards it, potentially beyond. Worth restricting to loopback or adding `-passwd`/`-rfbauth` if this hasn't been a deliberate choice.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/restart-browser]]
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
---
|
||||||
|
title: Hermes Agent
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-25
|
||||||
|
verified_on: 2026-07-25
|
||||||
|
confidence: high
|
||||||
|
tags: [system, hermes, agent, llm]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes Agent
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Primary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.
|
||||||
|
|
||||||
|
## Current Status
|
||||||
|
- **Version:** v0.19.0 (2026.7.20), upstream `9ecacd6b`
|
||||||
|
- **Install method:** git
|
||||||
|
- **Python:** 3.11.15
|
||||||
|
- **OpenAI SDK:** 2.24.0
|
||||||
|
- **Upstream status:** Up to date
|
||||||
|
- **Install directory:** `/home/hermes/.hermes/hermes-agent`
|
||||||
|
- **Community resources:** [Hermes Registry](https://babylondreams.de/hermes-registry/) — third-party plugins, skills, and extensions
|
||||||
|
- **Binary:** `/home/hermes/.local/bin/hermes`
|
||||||
|
- **Config:** `~/.hermes/config.yaml` (version 33)
|
||||||
|
- **Profile:** `default` (`~/.hermes/profiles/default/`)
|
||||||
|
|
||||||
|
## Location
|
||||||
|
- **Host:** Local Linux host (no containerized Hermes runtime confirmed)
|
||||||
|
- **Filesystem root:** `~/.hermes/`
|
||||||
|
- **Working directory:** `/home/hermes/workspace`
|
||||||
|
|
||||||
|
## Providers & Routing
|
||||||
|
- **Default provider:** `custom:litellm` → `http://litellm:4000/v1`
|
||||||
|
- **Fallback provider:** `custom:omniroute` → `http://omniroute:20128/v1`
|
||||||
|
- **Default model:** `freellmapi-auto`
|
||||||
|
- **Provider model count:** ~3430 (omniroute), ~1512 (litellm)
|
||||||
|
- **free-stack combo (OmniRoute):** Nemotron 3 Ultra 550B configured with `max_tokens: 8192`, `reasoning_effort: medium`, `chat_template_kwargs.force_nonempty_content: true` (see [[omniroute]])
|
||||||
|
- See [[model-providers]] for full breakdown.
|
||||||
|
|
||||||
|
## Memory
|
||||||
|
- **Backend:** Mnemosyne (local native)
|
||||||
|
- **Memory enabled:** true
|
||||||
|
- **Char limit:** 5000
|
||||||
|
- **User profile enabled:** true
|
||||||
|
- **User char limit:** 3000
|
||||||
|
- **Flush every:** 6 turns
|
||||||
|
- **Auto sleep:** true
|
||||||
|
- **Default scope:** session
|
||||||
|
- **Vector type:** int8
|
||||||
|
- **Shared surface:** `data/shared/mnemosyne.db`
|
||||||
|
- See [[memory-backend]] for details.
|
||||||
|
|
||||||
|
## Browser
|
||||||
|
- **Cloud provider:** Camoufox
|
||||||
|
- **Managed persistence:** true
|
||||||
|
- **Inactivity timeout:** 120 s
|
||||||
|
- **Allow private URLs:** true
|
||||||
|
- **cua-driver:** v0.9.0 at `/home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver`
|
||||||
|
- **DISPLAY:** `:102` (Xvfb/X11VNC/fluxbox confirmed working per session notes)
|
||||||
|
- See [[browser-backend]] for details.
|
||||||
|
|
||||||
|
## Terminal
|
||||||
|
- **Backend:** local
|
||||||
|
- **Working directory:** `/home/hermes/workspace`
|
||||||
|
- **Timeout:** 180 s
|
||||||
|
- **Home mode:** auto
|
||||||
|
- **Container CPU:** 1
|
||||||
|
- **Persistent shell:** true
|
||||||
|
- See [[terminal-backend]] for details.
|
||||||
|
|
||||||
|
## Skills
|
||||||
|
- **Total installed:** 66
|
||||||
|
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
||||||
|
- See [[skills-index]] for categorized list.
|
||||||
|
|
||||||
|
## Multi-Profile Multiplexer
|
||||||
|
|
||||||
|
The default gateway runs as a **profile multiplexer** (`gateway.multiplex_profiles: true`), serving both the `default` and `ops` profiles through a single gateway process with one Telegram connection.
|
||||||
|
|
||||||
|
### Profiles
|
||||||
|
|
||||||
|
| Profile | Chat | Purpose |
|
||||||
|
|---------|------|---------|
|
||||||
|
| default | `-1004321904721` (Rhino chat) | Main agent |
|
||||||
|
| ops | `-1003914987043` (Elephant ops), `-1003932503629` (OPS Tember rescue) | Operations/infra agent |
|
||||||
|
|
||||||
|
### How It Works
|
||||||
|
|
||||||
|
- **One gateway, one Telegram bot token** — shared across profiles
|
||||||
|
- **`profile_routes`** maps chat_id → profile at the gateway level
|
||||||
|
- **`allowed_chats` on both profiles** must include ALL chats that either profile should hear, because the default profile's adapter processes all inbound messages before `profile_routes` stamps `source.profile`
|
||||||
|
- The ops profile does NOT need `platforms.telegram.enabled` — the default adapter handles it
|
||||||
|
|
||||||
|
### Critical Rule: allowed_chats Is Pre-Route
|
||||||
|
|
||||||
|
The `allowed_chats` gate runs on the **default profile's adapter** before `profile_routes` matching. If a chat_id is only in the ops profile's `allowed_chats` but not the default's, messages from that chat are silently dropped before routing ever fires.
|
||||||
|
|
||||||
|
**Fix applied 2026-07-28:** Added both ops group IDs to the default profile's `allowed_chats` and `group_allowed_chats` (comma-separated). See [[multiplexer-setup]].
|
||||||
|
|
||||||
|
## Messaging Integrations
|
||||||
|
- **Telegram:** enabled, `allowed_chats: -1004321904721,-1003914987043,-1003932503629`, rich messages
|
||||||
|
- **Discord:** enabled, streaming true
|
||||||
|
- **Slack:** enabled
|
||||||
|
- **Email:** connected
|
||||||
|
- **Mattermost:** connected
|
||||||
|
- **ntfy:** connected
|
||||||
|
- **Google Chat:** available (`hermes-google_chat`)
|
||||||
|
- **QQ:** available (`hermes-qqbot`)
|
||||||
|
- **Yuanbao:** available
|
||||||
|
- See [[messaging-integrations]] for details.
|
||||||
|
|
||||||
|
## Scheduled Tasks
|
||||||
|
- **Cron jobs:** 25 active jobs
|
||||||
|
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||||
|
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher
|
||||||
|
- See [[scheduled-tasks]] for full table.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- LiteLLM proxy at `litellm:4000`
|
||||||
|
- OmniRoute at `omniroute:20128`
|
||||||
|
- cua-driver binary at path above
|
||||||
|
- Xvfb + X11VNC + fluxbox for headed browser
|
||||||
|
- Telegram bot token, Discord token, etc. (stored outside wiki)
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- Router migration incomplete
|
||||||
|
- Browser persistence not yet fully verified
|
||||||
|
- Obsidian Git plugin auto-sync caused conflicts during vault reorganization
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[model-providers]]
|
||||||
|
- [[memory-backend]]
|
||||||
|
- [[browser-backend]]
|
||||||
|
- [[terminal-backend]]
|
||||||
|
- [[skills-index]]
|
||||||
|
- [[messaging-integrations]]
|
||||||
|
- [[scheduled-tasks]]
|
||||||
|
- [[current-state]]
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
title: hermes-ui (Browser Hermes Desktop)
|
||||||
|
type: system
|
||||||
|
status: live
|
||||||
|
created: 2026-07-30
|
||||||
|
updated: 2026-07-30
|
||||||
|
verified_on: 2026-07-30
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, webui, desktop, browser]
|
||||||
|
---
|
||||||
|
|
||||||
|
# hermes-ui
|
||||||
|
|
||||||
|
The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo (`apps/desktop`) and repackaged as a plain Vite web app — runs in a browser instead of Electron.
|
||||||
|
|
||||||
|
- **Repo:** https://github.com/przbadu/hermes-ui
|
||||||
|
- **Upstream:** NousResearch/hermes-agent monorepo, commit `56a8e81d33a524f0ba0d68b6d54c8786ed283fb8` (2026-07-08)
|
||||||
|
- **Extraction date:** 2026-07-11
|
||||||
|
- **License:** MIT (Copyright (c) 2025 Nous Research)
|
||||||
|
|
||||||
|
## Deployments
|
||||||
|
|
||||||
|
| Host | Location | Port | Status | Started |
|
||||||
|
|------|----------|------|--------|---------|
|
||||||
|
| CT460 (this box) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active |
|
||||||
|
| CT450 (hermesagent) | `/home/hermes/hermes-ui/` | 5174 | Vite dev server | Active (since 2026-07-30) |
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
|
||||||
|
- Same-origin proxy via Vite so cookies and the WebSocket work without CORS issues.
|
||||||
|
- Electron-only methods are stubbed behind a capability flag in `src/web-bridge/`.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install deps (uses bun)
|
||||||
|
cd ~/hermes-ui/app && bun install
|
||||||
|
|
||||||
|
# Start dev server
|
||||||
|
cd ~/hermes-ui && bin/dev
|
||||||
|
|
||||||
|
# Or skip bin/dev and run vite directly:
|
||||||
|
bunx vite --port 5174 --host 0.0.0.0
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `PLAN.md` — extraction plan
|
||||||
|
- `UPSTREAM.md` — upstream commit diff guide
|
||||||
|
- `scripts/serve-on-gateway.sh` — serve built UI same-origin via `hermes serve`
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
title: Home Assistant
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: low
|
||||||
|
tags: [system, automation]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Home Assistant
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Home automation and device control; one of Hermes's messaging/integration platforms.
|
||||||
|
|
||||||
|
## Location
|
||||||
|
- **Verified (homelab infra docs, not this host):** VM 100 on `pve`, hostname `haos14.0` (Home Assistant OS), internal IP `172.30.232.1`.
|
||||||
|
- **Verified (Hermes config):** `homeassistant` and `hermes-homeassistant` appear in Hermes's platform/display integration lists (`~/.hermes/config.yaml`), alongside `qqbot`, `signal`, `slack`, `teams`, `telegram`.
|
||||||
|
- **Unknown:** the actual connection details (HA base URL, long-lived access token, which entities/automations Hermes can reach) are not present in the sanitized config snapshot — likely held in a plugin-specific credentials file not captured there, or not yet wired up beyond being enabled as a platform.
|
||||||
|
|
||||||
|
## Current configuration
|
||||||
|
Not verified beyond the platform being listed as available/connected in `current-state.md`'s "WebUI connected platforms" line. No automations, entity lists, or voice-assistant config confirmed from this host.
|
||||||
|
|
||||||
|
## Connected systems
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[infrastructure/proxmox]] — Runs as VM 100 on PVE
|
||||||
|
- [[systems/messaging-integrations]] — Integration platform
|
||||||
|
- [[systems/homepage-dashboard]] — Shown in dashboard More tab
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- Genuine gap, not a "known issue" per se: this page cannot confirm HA is *actively* used by Hermes vs. merely enabled — worth verifying directly (e.g. a live call to a known HA entity) next time someone is in this area.
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
None yet — would be a good candidate once the connection is verified (e.g. `runbooks/restart-home-assistant-bridge`).
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
---
|
||||||
|
title: Homepage Dashboard
|
||||||
|
type: systems
|
||||||
|
status: active
|
||||||
|
created: 2026-07-26
|
||||||
|
updated: 2026-07-26
|
||||||
|
confidence: high
|
||||||
|
tags: [dashboard, homepage, homelab, proxmox, monitoring]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Homepage Dashboard
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Central dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.
|
||||||
|
|
||||||
|
## URLs
|
||||||
|
- **Tailscale:** `https://runtipi.kangaroo-eel.ts.net:8443/`
|
||||||
|
- **Local:** `http://runtipi.kangaroo-eel.ts.net:8082/`
|
||||||
|
- **Manager UI:** `http://runtipi.kangaroo-eel.ts.net:8084/`
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
```
|
||||||
|
CT201 (runtipi)
|
||||||
|
├── Homepage container (port 8082) — main dashboard
|
||||||
|
├── Manager container (port 8084) — web UI for adding/removing apps
|
||||||
|
└── Tailscale serve (port 8443) — HTTPS proxy to Homepage
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tabs
|
||||||
|
| Tab | Content |
|
||||||
|
|---|---|
|
||||||
|
| Overview | Infrastructure (2 cols) + Everyday (4 cols) + Resources |
|
||||||
|
| Media | Jellyfin, arr stack, RSS |
|
||||||
|
| AI | OmniRoute, LiteLLM, Lobe Chat, SearXNG |
|
||||||
|
| Agents | Hermes, OpenClaw, Nanobot, Camofox |
|
||||||
|
| Tools | IT tools, Cyber Chef, etc. + Storage |
|
||||||
|
| More | Home Assistant, N8n, Jupyter |
|
||||||
|
| All Apps | 114 apps in 5-column grid (alphabetical) |
|
||||||
|
|
||||||
|
## Proxmox Integration
|
||||||
|
- **Node widget:** Shows live VM/LXC counts, CPU, memory for PVE node
|
||||||
|
- **Per-VM stats:** Click "View Proxmox stats" on mapped services
|
||||||
|
- **API token:** `homepage@pve!homepage` (privsep=0)
|
||||||
|
- **proxmox.yaml:** Config at `/opt/homepage/config/proxmox.yaml`
|
||||||
|
|
||||||
|
### VM/LXC Mapping
|
||||||
|
| Service | VMID | Type |
|
||||||
|
|---|---|---|
|
||||||
|
| PVE | (node) | — |
|
||||||
|
| Portainer docker | 200 | lxc |
|
||||||
|
| Portainer debianVM | 101 | qemu |
|
||||||
|
| Portainer openweb | 245 | lxc |
|
||||||
|
|
||||||
|
> Portainer on CT245 (`openweb.kangaroo-eel.ts.net:9443`) now uses a Tailscale/Let's Encrypt cert (installed 2026-08-09). Files: `/root/portainer-certs/portainer.{crt,key}` mounted into the container, `--sslcert/--sslkey`. Renewal: `/root/renew-portainer-cert.sh` via crontab (04:00 on the 1st monthly) — `tailscale cert --min-validity=20d` then `docker restart portainer`. Cert validity ~90 days.
|
||||||
|
| Runtipi | 201 | lxc |
|
||||||
|
| Gitea | 221 | lxc |
|
||||||
|
| FreshRSS | 260 | lxc |
|
||||||
|
| NextCloud | 270 | lxc |
|
||||||
|
| LiteLLM | 500 | lxc |
|
||||||
|
| OmniRoute | 600 | lxc |
|
||||||
|
| Ntfy | 241 | lxc |
|
||||||
|
| Home Assistant | 100 | qemu |
|
||||||
|
| N8n | 215 | lxc |
|
||||||
|
| Mattermost | 255 | lxc |
|
||||||
|
| SFTPGo | 299 | lxc |
|
||||||
|
| Cliproxy API | 103 | lxc |
|
||||||
|
| OpenClaw | 403 | qemu |
|
||||||
|
| Zeroclaw | 333 | lxc |
|
||||||
|
| Hermes | 460 | lxc |
|
||||||
|
| Hindsight | 450 | lxc |
|
||||||
|
|
||||||
|
## Configuration Files
|
||||||
|
All at `/opt/homepage/config/` on CT201:
|
||||||
|
- `services.yaml` — All apps, groups, and Proxmox mappings
|
||||||
|
- `settings.yaml` — Theme, layout, tabs
|
||||||
|
- `custom.css` — Warm amber accent overrides
|
||||||
|
- `widgets.yaml` — System health, clock
|
||||||
|
- `proxmox.yaml` — PVE API token
|
||||||
|
- `docker.yaml` — Docker socket integration
|
||||||
|
|
||||||
|
## Manager UI
|
||||||
|
Web form at port 8084 for adding/removing apps without SSH.
|
||||||
|
- **Add:** Name, Category (dropdown), URL, optional Icon
|
||||||
|
- **Delete:** Search apps, click Delete
|
||||||
|
- **Auto-restart:** Homepage restarts after each change
|
||||||
|
|
||||||
|
## Quick Launch Search
|
||||||
|
Type anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps.
|
||||||
|
|
||||||
|
## Theme
|
||||||
|
- Dark base with warm amber (#d4a04a) accents
|
||||||
|
- Bigger tiles (68px min-height, 40px icons)
|
||||||
|
- Custom CSS in `custom.css`
|
||||||
|
|
||||||
|
## Tailscale Serve
|
||||||
|
Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.
|
||||||
|
|
||||||
|
## Maintenance
|
||||||
|
- **Restart Homepage:** `docker restart homepage`
|
||||||
|
- **Restart Manager:** `docker restart homepage-manager`
|
||||||
|
- **Edit apps manually:** Edit `/opt/homepage/config/services.yaml` then restart
|
||||||
|
- **Docker auto-discovery:** Add `homepage.group`, `homepage.name`, `homepage.href` labels to containers
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
title: Memory Backend
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-01
|
||||||
|
verified_on: 2026-08-01
|
||||||
|
confidence: high
|
||||||
|
tags: [memory, mnemosyne, agent]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Memory Backend
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Persist and recall agent memory across sessions.
|
||||||
|
|
||||||
|
## Current Configuration
|
||||||
|
- **Primary backend:** Mnemosyne (local native)
|
||||||
|
- **Memory enabled:** true
|
||||||
|
- **User profile enabled:** true
|
||||||
|
- **Memory char limit:** 5000
|
||||||
|
- **User char limit:** 3000
|
||||||
|
- **Flush every:** 6 turns
|
||||||
|
- **Auto sleep:** true
|
||||||
|
- **Default scope:** session
|
||||||
|
- **Vector type:** int8
|
||||||
|
- **Profile isolation:** true (configured; requires gateway restart for the current messaging runtime)
|
||||||
|
- **Shared surface:** `data/shared/mnemosyne.db`
|
||||||
|
- **Shared surface read:** true
|
||||||
|
- **Sync roles:** user
|
||||||
|
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||||||
|
- **Reflect max calls per session:** 3
|
||||||
|
|
||||||
|
## Storage Location
|
||||||
|
- Mnemosyne home: `~/.hermes/mnemosyne/`
|
||||||
|
- DB: `~/.hermes/mnemosyne/data/`
|
||||||
|
- Backups: `~/.hermes/mnemosyne/backups/`
|
||||||
|
- Logs: `~/.hermes/mnemosyne/logs/`
|
||||||
|
- Models: `~/.hermes/mnemosyne/models/`
|
||||||
|
- Config: `~/.hermes/mnemosyne/config.yaml`
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- Health monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.
|
||||||
|
- The 2026-08-01 automatic-capture failure and repair are documented in [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
|
|
||||||
|
**Correction (2026-07-22, per Anthony):** this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see [[comparisons/memory-backends]].
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[memory-backend]]
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[experiments/index]]
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
title: Messaging Integrations
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [messaging, telegram, discord, email, integrations]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Messaging Integrations
|
||||||
|
|
||||||
|
## Telegram
|
||||||
|
- **Status:** enabled, active
|
||||||
|
- **allowed_chats:** `-1004321904721` (primary)
|
||||||
|
- **Rich messages:** true
|
||||||
|
- **Platform toolsets:** cli, telegram (full suite)
|
||||||
|
- **Delivery targets seen in cron:** telegram, `telegram:1793951355`, `telegram:-1004321904721:1030`, `telegram:-1004321907D2:346`, `telegram:-1004321904721:273`
|
||||||
|
- **Notes:** Primary delivery target for scheduled jobs
|
||||||
|
|
||||||
|
## Discord
|
||||||
|
- **Status:** enabled, streaming true
|
||||||
|
- **Platform:** `hermes-discord`
|
||||||
|
- **Platform toolsets:** cli, discord
|
||||||
|
|
||||||
|
## Email
|
||||||
|
- **Status:** connected
|
||||||
|
- **Libraries:** himalaya CLI available
|
||||||
|
- **Platform toolsets:** cli
|
||||||
|
|
||||||
|
## Slack
|
||||||
|
- **Status:** available (toolsets: cli, slack-platform)
|
||||||
|
- **Config noted:** slack entries in config but Slack enabled status unclear from config grep
|
||||||
|
|
||||||
|
## Other Platforms Available
|
||||||
|
- **Mattermost:** toolsets present (`hermes-mattermost`)
|
||||||
|
- **Google Chat:** toolsets present (`hermes-google_chat`)
|
||||||
|
- **QQ:** toolsets present (`hermes-qqbot`)
|
||||||
|
- **Signal:** toolsets present (`hermes-signal`)
|
||||||
|
- **WhatsApp:** toolsets present (`hermes-whatsapp`)
|
||||||
|
- **ntfy:** noted in memory config outputs but runtime status unverified
|
||||||
|
- **Yuanbao:** toolsets present (`hermes-yuanbao`)
|
||||||
|
- **Home Assistant:** homeassistant-platform enabled
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- Telegram thread_id `34795` for `telegram:1793951355` not found — one cron job delivered without thread_id
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[messaging-integrations]]
|
||||||
|
- [[scheduled-tasks]]
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
title: Mnemosyne
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-02
|
||||||
|
verified_on: 2026-08-02
|
||||||
|
confidence: high
|
||||||
|
tags: [memory, mnemosyne, backend]
|
||||||
|
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mnemosyne
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Primary agent memory store for Hermes.
|
||||||
|
|
||||||
|
## Current Configuration
|
||||||
|
- **Provider:** `mnemosyne`
|
||||||
|
- **Enabled:** true
|
||||||
|
- **Auto sleep:** true
|
||||||
|
- **Default scope:** `session`
|
||||||
|
- **Memory char limit:** 5000
|
||||||
|
- **User char limit:** 3000
|
||||||
|
- **Vector type:** int8
|
||||||
|
- **Profile isolation:** true
|
||||||
|
- **Shared surface read:** true
|
||||||
|
- **Shared surface path:** `data/shared/mnemosyne.db`
|
||||||
|
- **Skip contexts:** cron, flush, subagent, background, skill_loop
|
||||||
|
- **Reflect max calls per session:** 3
|
||||||
|
- **Sync roles:** user
|
||||||
|
- **Flush every:** 6 turns
|
||||||
|
- **Embedding model:** `BAAI/bge-small-en-v1.5` (384 dims)
|
||||||
|
- **Embeddings via API:** false (local fastembed)
|
||||||
|
|
||||||
|
## Database Stats (2026-08-02)
|
||||||
|
| Table | Rows | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `working_memory` | 1,476 | Active memories |
|
||||||
|
| `episodic_memory` | 458 | Consolidated summaries |
|
||||||
|
| `memory_embeddings` | 158 | Only working memories — **0 episodic** |
|
||||||
|
| `memory` (legacy) | 2,181 | Old table, not migrated |
|
||||||
|
| `vec_working` | 159 | Vector index rows |
|
||||||
|
| Shared surface | 0 | Empty — cross-agent sharing not functional |
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
|
||||||
|
### Vector embedding coverage gap
|
||||||
|
- **Only 159/1,476 working memories (10.8%)** have vector embeddings
|
||||||
|
- **0/458 episodic memories** have embeddings
|
||||||
|
- Semantic search (`mnemosyne_recall`) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories
|
||||||
|
- This means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store
|
||||||
|
- **Root cause:** The embedding pipeline (fastembed) never processed the bulk of memories. The `vec_working` table was rebuilt once (Jul 18) but only captured 159 rows.
|
||||||
|
- **Fix needed:** Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.
|
||||||
|
|
||||||
|
### Profile isolation chosen over shared surface
|
||||||
|
|- The ops profile was enabled with mnemosyne on 2026-08-02
|
||||||
|
|- **Decision:** `profile_isolation: true` was set on ops, giving each profile its own independent memory bank — zero cross-contamination
|
||||||
|
|- `data/shared/mnemosyne.db` exists with 0 rows; cross-profile sharing is intentionally unused
|
||||||
|
|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour
|
||||||
|
|- Config in `~/.hermes/profiles/ops/config.yaml`: `memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|
||||||
|
|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently
|
||||||
|
|
||||||
|
### Legacy table not migrated
|
||||||
|
- 2,181 rows in the old `memory` table are not in the new `working_memory`/`episodic_memory` system
|
||||||
|
- These are only accessible via the legacy `memory` tool, not via `mnemosyne_recall`
|
||||||
|
|
||||||
|
## Known Management Jobs
|
||||||
|
- Mnemosyne Auto-Consolidate every 30m — script: `consolidate_mnemosyne.sh`, mode: no-agent
|
||||||
|
- Mnemosyne DB Backup weekly Sunday 03:00 — script: `mnemosyne-backup.sh`, mode: no-agent
|
||||||
|
- Mnemosyne Health Monitor every 12h — skill: `mnemosyne-operations`, reports only on exception
|
||||||
|
- Mnemosyne Watchdog every 15m — script: `mnemosyne-watchdog.py`, mode: no-agent
|
||||||
|
|
||||||
|
## History and Evidence
|
||||||
|
- **2026-08-01 incident:** Automatic conversational capture was broken by stringified structured YAML, especially `sync_roles: "['user']"`. Fixed by rewriting with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||||||
|
- **2026-08-01 cleanup:** Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).
|
||||||
|
- **2026-07-29 incident:** `memory.mnemosyne.tools: None` caused schema loading warnings. Fixed by removing the line. See [[incidents/2026-07-29-mnemosyne-relay]].
|
||||||
|
- **2026-07-29 relay scope corruption:** `RuntimeError: scope handle is not at the top of the stack` — 3 occurrences, self-resolved.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[concepts/agent-memory]]
|
||||||
|
- [[comparisons/memory-backends]]
|
||||||
|
- [[incidents/2026-08-01-mnemosyne-capture-failure]]
|
||||||
|
- [[incidents/2026-07-29-mnemosyne-relay]]
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
---
|
||||||
|
title: Model Providers
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-14
|
||||||
|
verified_on: 2026-08-14
|
||||||
|
confidence: high
|
||||||
|
tags: [model, provider, inference, routing]
|
||||||
|
sources: [live-probe, config, web-search]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Model Providers
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Inference providers available to Hermes, their routing paths, and notable model updates.
|
||||||
|
|
||||||
|
## Current Provider Stack
|
||||||
|
| Provider | URL | Role |
|
||||||
|
|---|---|---|
|
||||||
|
| `custom:litellm` | `http://litellm:4000/v1` | Main inference (slower path, 6–26s/call) |
|
||||||
|
| `custom:omni` | `http://omni:20128/v1` | Auxiliary roles (vision, skills, approval, mcp, web_extract, compression) — factory-reset 2026-08-14 |
|
||||||
|
| `opencode-zen` | (internal) | Free-tier model access for cron jobs |
|
||||||
|
| Guanaco | `ollama:11434` | LLM proxy/router on CT205 (Ollama stack) |
|
||||||
|
|
||||||
|
## Default Model
|
||||||
|
- `resilient-free` — resolves through OmniRoute (changed from `auto/best-free` on 2026-08-14 after factory reset)
|
||||||
|
- Previous default `auto/best-free` was misconfigured — had no combo definition and routed to paid `gpt-5.6-sol`
|
||||||
|
- `freellmapi-auto` (LiteLLM path) still available but slower
|
||||||
|
|
||||||
|
## Notable Model Updates
|
||||||
|
|
||||||
|
### GPT-5.6 Luna — Model Research (2026-08-14)
|
||||||
|
- **Community consensus:** Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper
|
||||||
|
- **Luna strengths:** visual/UI work, final code review, bounded high-stakes tasks, cleaner structured output
|
||||||
|
- **DeepSeek strengths:** automation, persistent executor work, high-volume subtasks, long-context economics
|
||||||
|
- **Key caveat:** Provider route matters more than model name — `openai-codex/gpt-5.6-luna`, `openrouter/openai/gpt-5.6-luna`, `deepseek/deepseek-v4-flash`, and `openrouter/deepseek/deepseek-v4-flash` are four different tests
|
||||||
|
- **Pricing:** Direct OpenAI $0.20/M input, $1.20/M output; OpenRouter $0.10/M input, $0.60/M output
|
||||||
|
- **Context:** 1.05M tokens, 128K output, function calling, structured outputs, image input, configurable reasoning effort
|
||||||
|
- **Recommendation:** Luna plans and reviews; DeepSeek executes — but test on a clean direct route before judging
|
||||||
|
|
||||||
|
### Guanaco v0.8.14 (2026-08-14)
|
||||||
|
- Updated from v0.8.11 on CT205 — three incremental bug-fix releases focused on DSML/tool-call output hygiene
|
||||||
|
- DSML tag leakage fixes: malformed tool-call output no longer leaks raw syntax junk into model context
|
||||||
|
- Dashboard performance fix: cached content size, fixed wrong column names in history query
|
||||||
|
- No new features, no breaking changes — low-risk update
|
||||||
|
- See [[systems/guanaco]]
|
||||||
|
|
||||||
|
### DeepSeek V4 Flash 0731 (2026-07-31)
|
||||||
|
- **Same model ID:** `deepseek-v4-flash` — no ID change, just a post-training bump
|
||||||
|
- **Same architecture:** 284B total params, 13B active per token, 1M context window
|
||||||
|
- **Same pricing:** $0.14/M input, $0.28/M output
|
||||||
|
- **Benchmark jump:** Terminal Bench 2.1: 82.7 (up from 61.8 preview — 34% relative gain)
|
||||||
|
- **Now surpasses V4-Pro preview** (72.1) on agent/coding benchmarks
|
||||||
|
- **Stricter tool name validation:** Rejects tool names not matching `^[a-zA-Z0-9_-]+$` — may cause issues with OmniRoute-transformed tool names
|
||||||
|
- **Impact:** Your existing cron jobs using `deepseek-v4-flash` get the upgrade automatically. The empty-stream fallback issue (`opencode-go/deepseek-v4-flash` returning empty 200s) is a provider plumbing problem, not a model quality problem.
|
||||||
|
|
||||||
|
### OmniRoute Free-DeepSeek Silent-Stop Mitigation (2026-08-04)
|
||||||
|
- Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
|
||||||
|
- Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
|
||||||
|
- See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
|
||||||
|
|
||||||
|
## Context Enforcement
|
||||||
|
- Three layers: model native → combo `data.config.context_length` → Hermes client cap
|
||||||
|
- Effective context is the minimum of the three
|
||||||
|
|
||||||
|
## Combo Behavior
|
||||||
|
- `priority` retries same provider up to `maxRetries`, then errors; it does NOT fall through
|
||||||
|
- `fill-first` walks the list on failure
|
||||||
|
- `auto`/`lkgp` recalculates per request using health scoring
|
||||||
|
- Same-provider entries in a combo share rate-limit buckets
|
||||||
|
- Session affinity caches provider in `session_model_history`
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[concepts/model-routing]]
|
||||||
|
- [[systems/omniroute]]
|
||||||
|
- [[systems/opencode-go]]
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
title: Nanobot
|
||||||
|
type: system
|
||||||
|
status: degraded
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-07
|
||||||
|
verified_on: 2026-08-07
|
||||||
|
confidence: high
|
||||||
|
tags: [system, nanobot, network]
|
||||||
|
sources: [live-probe, omniroute-investigation]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nanobot
|
||||||
|
## Purpose
|
||||||
|
Role in Anthony's setup.
|
||||||
|
## Location
|
||||||
|
- **Verified 2026-07-22:** LXC 333, hostname `clawtest`, Tailscale IP `100.122.189.81`.
|
||||||
|
## Current state
|
||||||
|
- **Status: DOWN (as of 2026-08-07).** Connection refused on port 8900 — the OmniRoute provider connection to nanobot (`58debd59`, prefix `nanobot`) was causing event-loop stalls because every model sync + credential health check retried it. Connection disabled in OmniRoute. Nanobot on CT333 is not responding.
|
||||||
|
- **Prior state:** `nanobot` process listening on port `8766` (verified via `ss -tlnp` on ct333, 2026-07-22).
|
||||||
|
- `whatsapp-bridge.service` (systemd, active) — "WhatsApp Bridge for Nanobot" — also runs on ct333. Status unknown as of 2026-08-07.
|
||||||
|
- Publicly reachable via the same Cloudflare Tunnel as Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, `nano-ssh.martinwa.org` → SSH on the same host. See [[infrastructure/domains-and-tunnels]].
|
||||||
|
|
||||||
|
## Current state as of 2026-07-22 (superseded — see below)
|
||||||
|
~~- No nanobot process running on this host~~
|
||||||
|
~~- `nanobot` systemd user service: inactive~~
|
||||||
|
~~- Not present in `pgrep` process list~~
|
||||||
|
|
||||||
|
**Correction (this audit, same day):** the above was true only for *this* host (ct460) — Nanobot was never expected to run here. It **is** actively running, on LXC 333 ("clawtest"), confirmed via direct process/port check. Left the original lines struck through rather than deleted, per SCHEMA rule 9 (mark, don't silently delete).
|
||||||
|
|
||||||
|
## History
|
||||||
|
- **User-provided:** Nanobox on the PVE host at `192.168.178.39`
|
||||||
|
- **Correction:** `192.168.178.39` is `pve`'s own LAN IP (matches the Cloudflare Tunnel's `pve.martinwa.org` entry, which proxies to Proxmox's web UI on that address) — this was a conflation, not a second Nanobot instance. The real Nanobot host is ct333 (see Location above).
|
||||||
|
- **Verified:** Lan/Wi-Fi node at `192.168.178.46` running ARM/arm64 firmware; `ubnt` login at `192.168.1.1` does not apply to this node
|
||||||
|
- **Verified:** DWARF01 identified as a UniFi device; ONT01 status unresolved; DNS provided by `192.168.178.39`, not the router
|
||||||
|
- **User-provided:** Avoid `heroku.com/archive` except for CAPI; use official CAPI course downloads instead
|
||||||
|
- **Superseded:** "Nanobot may have been used previously as Hermes gateway or tunnel front on another host" / "Unknown: whether nanobot should be redeployed" — no longer applicable now that it's confirmed running; the open question is now *why* it's running alongside Hermes and whether that's intentional (see [[comparisons/hermes-vs-nanobot]]).
|
||||||
|
|
||||||
|
## Connected systems
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/headless-server]]
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- Overlap/purpose vs. Hermes not documented anywhere — see [[comparisons/hermes-vs-nanobot]] for the open question this raises.
|
||||||
|
## Change history
|
||||||
|
- 2026-07-22: audit corrected "not running" claim after direct verification on ct333.
|
||||||
|
- 2026-08-07: status changed to DEGRADED — nanobot confirmed DOWN (connection refused on 8900). OmniRoute provider connection disabled. CT333 service status unknown.
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[runbooks/restart-hermes]]
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[comparisons/hermes-vs-nanobot]]
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
---
|
||||||
|
title: Obsidian Vault
|
||||||
|
type: systems
|
||||||
|
status: active
|
||||||
|
created: 2026-07-26
|
||||||
|
updated: 2026-07-26
|
||||||
|
confidence: high
|
||||||
|
tags: [obsidian, notes, vault, homelab]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Obsidian Vault
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Personal knowledge base — notes, research, projects, and agent documentation.
|
||||||
|
|
||||||
|
## Details
|
||||||
|
- **Location:** `/home/hermes/wiki/` (on CT460)
|
||||||
|
- **Git repo:** Gitea (`Anthony/obsidian-vault`)
|
||||||
|
- **Structure:** NUMBERED folders (00-99)
|
||||||
|
- **Sync:** Git-based (manual or cron-triggered)
|
||||||
|
|
||||||
|
## Folder Structure
|
||||||
|
| Folder | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `00 - Home` | Dashboard and entry points |
|
||||||
|
| `10 - Inbox` | New/unprocessed notes |
|
||||||
|
| `20 - Agent Notes` | Notes from AI agents |
|
||||||
|
| `30 - Projects` | Active project documentation |
|
||||||
|
| `50 - Research` | Research and investigations |
|
||||||
|
| `60 - Resources` | Reference materials |
|
||||||
|
| `70 - Reference` | Quick lookup |
|
||||||
|
| `90 - Archive` | Completed/old material |
|
||||||
|
| `99 - Templates` | Note templates |
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
- Numbered folders preferred (Anthony corrected twice)
|
||||||
|
- Check Mnemosyne before vault work
|
||||||
|
- Don't impose structure changes without understanding usage
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[infrastructure/hosts]] — CT460 host details
|
||||||
|
- [[systems/memory-backend]] — Agent memory integration
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
title: OmniRoute
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-14
|
||||||
|
verified_on: 2026-08-14
|
||||||
|
confidence: high
|
||||||
|
tags: [routing, provider, model, inference]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# OmniRoute
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Secondary provider/router for Hermes auxiliary roles and some agent routing.
|
||||||
|
|
||||||
|
## Current Configuration
|
||||||
|
- **Hermes provider name:** `custom:omni`
|
||||||
|
- **Base URL:** `http://omni:20128/v1`
|
||||||
|
- **Host reachability:** `omni.kangaroo-eel.ts.net` resolves to `100.93.204.4`; CT601 running, direct HTTP reachable
|
||||||
|
- **Runs on:** Proxmox CT601 (`omni`) — migrated from CT600 (`omniroute`, now stopped) on 2026-08-13
|
||||||
|
- **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`)
|
||||||
|
- **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal`
|
||||||
|
- **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead
|
||||||
|
- **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp`
|
||||||
|
- **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki
|
||||||
|
|
||||||
|
## Catalog and Routing Facts
|
||||||
|
- Catalog reachable: yes, 47 combos returned at `/api/combos`
|
||||||
|
- Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid`
|
||||||
|
- Combo-level context can override individual models
|
||||||
|
- `auto`/`lkgp` strategies may silently change chosen models
|
||||||
|
- Model mappings can substitute models silently
|
||||||
|
- Session affinity caches last successful provider in `session_model_history`
|
||||||
|
|
||||||
|
## Catalog Evidence (2026-07-22)
|
||||||
|
- Advertised context for `gemini/gemini-3.6-flash`: `1048576`
|
||||||
|
- Related models in catalog: `kc/google/gemini-3.6-flash`, `kilocode/google/gemini-3.6-flash`, `nous/google/gemini-3.6-flash`, `opencode-zen/gemini-3.6-flash`, `vertex/gemini-3.6-flash`
|
||||||
|
- No live combo in the returned 47 combos referenced `gemini-3.6-flash`
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- opencode-zen proxy pitfall documented in omniroute-ops skill
|
||||||
|
- Web search proxy failures documented in omniroute-ops skill
|
||||||
|
- Combo discovery can differ from resolution;
|
||||||
|
|
||||||
|
## AutoRoute variants (2026-08-12)
|
||||||
|
- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
|
||||||
|
- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
|
||||||
|
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
|
||||||
|
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-14)
|
||||||
|
- **OmniRoute factory reset on CT601.** Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (`auto/best-fast`, `auto/best-coding`, etc.). Old state backed up at `/opt/omniroute/factory-reset-backup-20260814-225524`.
|
||||||
|
- **Providers re-added from Notion API vault:** Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.
|
||||||
|
- **Hermes config default model changed:** `auto/best-free` → `resilient-free` (via `~/.hermes/config.yaml`). `auto/best-free` was misconfigured — had no combo definition and routed to `gpt-5.6-sol` (paid) instead of free models. Switched to `resilient-free` which is a working free combo.
|
||||||
|
- **Multiple free combos broken by upstream model ID changes:**
|
||||||
|
- OpenRouter: `poolside/laguna-s-2-1:free` (dash) → `poolside/laguna-s-2.1:free` (dot) — model ID format changed
|
||||||
|
- Groq: `meta-llama/llama-4-scout-17b-16e-instruct` — **retired**, only llama-3.3/3.1 remain
|
||||||
|
- Affected combos: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`
|
||||||
|
- All route through OpenRouter → fail with 400/404 when trying these model IDs
|
||||||
|
- **Working free combos as of 2026-08-14:** `free-lkgp` (DeepSeek V4 Flash), `mega-free` (StepFun 3.7 Flash), `smart-route` / `smart-route-ia` (nested combos)
|
||||||
|
- **LiteLLM routing issue identified:** Hermes was previously pointing at `http://litellm:4000/v1` with `free-auto` model, causing 6–26s per call. OmniRoute's `auto/best-free` tested at 265ms. Config now points at OmniRoute.
|
||||||
|
- **GPT-5.6 Luna research:** Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-13)
|
||||||
|
- **OmniRoute migrated CT600 → CT601.** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. Database path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`. API key unchanged. All 14 config + 22 cron-job `custom:omniroute` references updated to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. Verified live 2026-08-13 (`pct list`: 600 stopped, 601 running; `grep`: 0 remaining `custom:omniroute` refs).
|
||||||
|
|
||||||
|
## Recent Changes (2026-07-25)
|
||||||
|
- **free-stack combo updated:** Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:
|
||||||
|
- `max_tokens`: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)
|
||||||
|
- `reasoning_effort`: medium (was high, reducing long internal monologues)
|
||||||
|
- `chat_template_kwargs.force_nonempty_content`: true (prevents empty responses when tool calling + reasoning)
|
||||||
|
- **Combo ID:** `23d25f8f-5eb0-4b6f-b966-e260ca3905b3`
|
||||||
|
- **Hermes config.yaml updated:** `agent.reasoning_effort: medium`, `agent.max_tokens: 8192`, `agent.chat_template_kwargs.force_nonempty_content: true`
|
||||||
|
- **Rationale:** NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-07)
|
||||||
|
- **Three-layer root cause for slowness identified and fixed:**
|
||||||
|
1. **npm package .env override:** OmniRoute loads three .env files (`/var/lib/omniroute/.env`, `/root/.omniroute/.env`, `/usr/lib/node_modules/omniroute/.env`). The npm package's `.env` ships with `ENABLE_SOCKS5_PROXY=true` and `npm update` re-installs it with that default, silently overriding the `/var/lib` config. Fix: flip all three to `false`, then restart. After ANY upgrade, check all three.
|
||||||
|
2. **Dead provider connection:** Connection `58debd59` (prefix `nanobot`) points at `clawtest:8900/v1` — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.
|
||||||
|
3. **PVE host load leaking into CT600:** CT600 has `nesting=1`, so `uptime`/`top` inside the container mirror PVE's host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from `pct exec 600 -- ps -o %cpu` or host-side `ps`, never from in-container load average.
|
||||||
|
- **First-request-after-cache-expiry stall:** `/v1/models` hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don't judge health on one curl — run a burst of 5 and check if later tries are sub-second.
|
||||||
|
- **`is_active=0` does NOT stop CredentialHealth probes:** Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs `[ProxyFetch] ECONNREFUSED` noise. That noise is benign once the endpoint is unreachable.
|
||||||
|
- **Lessons banked in `omniroute-ops` skill:** `references/omniroute-upgrade-procedure.md` updated with all three pitfalls.
|
||||||
|
|
||||||
|
## Recent Changes (2026-08-04)
|
||||||
|
- **OmniRoute free-DeepSeek silent-stop mitigation:** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[model-providers]]
|
||||||
|
- [[model-routing]]
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
---
|
||||||
|
title: OpenCode Go
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-29
|
||||||
|
updated: 2026-07-29
|
||||||
|
verified_on: 2026-07-29
|
||||||
|
confidence: high
|
||||||
|
tags: [opencode, provider, models, subscription, limits]
|
||||||
|
sources: [https://opencode.ai/docs/go/]
|
||||||
|
---
|
||||||
|
|
||||||
|
# OpenCode Go
|
||||||
|
|
||||||
|
Low cost subscription for open coding models — $5 first month, then **$10/month**. Designed for international users, models hosted in US, EU, and Singapore.
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
1. Sign in to [OpenCode Zen](https://opencode.ai/zen), subscribe to Go, copy your API key
|
||||||
|
2. Run `/connect` in the TUI, select `OpenCode Go`, paste your API key
|
||||||
|
3. Run `/models` to see available models
|
||||||
|
|
||||||
|
> Only one member per workspace can subscribe.
|
||||||
|
|
||||||
|
## Usage Limits
|
||||||
|
|
||||||
|
| Window | Dollar Limit |
|
||||||
|
|--------|-------------|
|
||||||
|
| 5 hours | $12 |
|
||||||
|
| Weekly | $30 |
|
||||||
|
| Monthly | $60 |
|
||||||
|
|
||||||
|
Limits are in **dollar value**, so your actual request count depends on the model. Cheaper models (DeepSeek V4 Flash, MiMo-V2.5) get far more requests than expensive ones (Kimi K3, Grok 4.5).
|
||||||
|
|
||||||
|
### Estimated Requests Per Window
|
||||||
|
|
||||||
|
Based on typical Go usage patterns:
|
||||||
|
|
||||||
|
| Model | Per 5h | Per Week | Per Month |
|
||||||
|
|-------|--------|----------|-----------|
|
||||||
|
| Grok 4.5 | 120 | 300 | 600 |
|
||||||
|
| GLM-5.2 | 880 | 2,150 | 4,300 |
|
||||||
|
| GLM-5.1 | 880 | 2,150 | 4,300 |
|
||||||
|
| Kimi K3 | 110 | 250 | 490 |
|
||||||
|
| Kimi K2.7 Code | 1,350 | 3,380 | 6,750 |
|
||||||
|
| Kimi K2.6 | 1,150 | 2,880 | 5,750 |
|
||||||
|
| MiMo-V2.5 | 30,100 | 75,200 | 150,400 |
|
||||||
|
| MiMo-V2.5-Pro | 3,250 | 8,150 | 16,300 |
|
||||||
|
| MiniMax M3 | 3,200 | 8,000 | 16,000 |
|
||||||
|
| MiniMax M2.7 | 3,400 | 8,500 | 17,000 |
|
||||||
|
| Qwen3.7 Max | 950 | 2,390 | 4,770 |
|
||||||
|
| Qwen3.7 Plus | 4,300 | 10,800 | 21,600 |
|
||||||
|
| Qwen3.6 Plus | 3,300 | 8,200 | 16,300 |
|
||||||
|
| DeepSeek V4 Pro | 3,450 | 8,550 | 17,150 |
|
||||||
|
| DeepSeek V4 Flash | 31,650 | 79,050 | 158,150 |
|
||||||
|
| Hy3 | 4,300 | 10,750 | 21,500 |
|
||||||
|
|
||||||
|
### Token Estimates Per Request
|
||||||
|
|
||||||
|
| Model | Input Tokens | Cached Tokens | Output Tokens |
|
||||||
|
|-------|-------------|---------------|---------------|
|
||||||
|
| Grok 4.5 | 1,100 | 71,500 | 220 |
|
||||||
|
| GLM-5.2/5.1 | 700 | 52,000 | 150 |
|
||||||
|
| Kimi K3 | 1,050 | 76,500 | 300 |
|
||||||
|
| Kimi K2.7/K2.6 | 870 | 55,000 | 200 |
|
||||||
|
| DeepSeek V4 Pro | 750 | 82,000 | 290 |
|
||||||
|
| DeepSeek V4 Flash | 790 | 68,000 | 280 |
|
||||||
|
| MiniMax M3 | 510 | 56,000 | 190 |
|
||||||
|
| MiniMax M2.7 | 300 | 55,000 | 125 |
|
||||||
|
| MiMo-V2.5 | 830 | 71,500 | 295 |
|
||||||
|
| MiMo-V2.5-Pro | 790 | 86,000 | 305 |
|
||||||
|
| Qwen3.7 Max | 420 | 66,000 | 200 |
|
||||||
|
| Qwen3.7 Plus | 500 | 57,000 | 190 |
|
||||||
|
| Qwen3.6 Plus | 500 | 57,000 | 190 |
|
||||||
|
| Hy3 | 830 | 71,500 | 295 |
|
||||||
|
|
||||||
|
### Per-Model Pricing & Monthly Usage Included
|
||||||
|
|
||||||
|
| Model | Input $/1M | Output $/1M | Cached Read $/1M | Monthly Usage |
|
||||||
|
|-------|-----------|------------|-----------------|---------------|
|
||||||
|
| Grok 4.5 | $2.00 | $6.00 | $0.30 | $15 |
|
||||||
|
| GLM-5.2 | $1.40 | $4.40 | $0.26 | $60 |
|
||||||
|
| GLM-5.1 | $1.40 | $4.40 | $0.26 | $60 |
|
||||||
|
| Kimi K3 | $3.00 | $15.00 | $0.30 | $15 |
|
||||||
|
| Kimi K2.7 Code | $0.95 | $4.00 | $0.19 | $60 |
|
||||||
|
| Kimi K2.6 | $0.95 | $4.00 | $0.16 | $60 |
|
||||||
|
| MiMo V2.5 | $0.14 | $0.28 | $0.0028 | $60 |
|
||||||
|
| MiMo V2.5 Pro | $0.435 | $0.87 | $0.0036 | $15 |
|
||||||
|
| MiniMax M3 | $0.30 | $1.20 | $0.06 | $60 |
|
||||||
|
| MiniMax M2.7 | $0.30 | $1.20 | $0.06 | $60 |
|
||||||
|
| Qwen3.7 Max | $2.50 | $7.50 | $0.50 | $60 |
|
||||||
|
| Qwen3.7 Plus (≤256K) | $0.40 | $1.60 | $0.04 | $60 |
|
||||||
|
| Qwen3.7 Plus (>256K) | $1.20 | $4.80 | $0.12 | $60 |
|
||||||
|
| Qwen3.6 Plus (≤256K) | $0.50 | $3.00 | $0.05 | $60 |
|
||||||
|
| Qwen3.6 Plus (>256K) | $2.00 | $6.00 | $0.20 | $60 |
|
||||||
|
| DeepSeek V4 Pro | $0.435 | $0.87 | $0.0036 | $15 |
|
||||||
|
| DeepSeek V4 Flash | $0.14 | $0.28 | $0.0028 | $60 |
|
||||||
|
| Hy3 | $0.14 | $0.58 | $0.035 | $60 |
|
||||||
|
|
||||||
|
Track usage in the [console](https://opencode.ai/zen).
|
||||||
|
|
||||||
|
> If you reach the usage limit, you can continue using the free models.
|
||||||
|
|
||||||
|
## Usage Beyond Limits
|
||||||
|
|
||||||
|
If you also have credits on your Zen balance, enable the **Use balance** option in the console. Go will fall back to your Zen balance after reaching usage limits instead of blocking requests.
|
||||||
|
|
||||||
|
## Why Some Models Have Lower Usage
|
||||||
|
|
||||||
|
With Go, you pay $10/month and they aim to give you **6x that in usage** ($60). For most models, this works through bulk discounts and reserved GPU capacity. For some models (Grok 4.5, Kimi K3, MiMo V2.5 Pro, DeepSeek V4 Pro), they haven't negotiated discounts yet — so the usage multiplier is lower (you get a little more than paying directly, just not the full 6x).
|
||||||
|
|
||||||
|
## Available Models
|
||||||
|
|
||||||
|
| Model | Model ID | Endpoint | SDK Package |
|
||||||
|
|-------|----------|----------|-------------|
|
||||||
|
| Grok 4.5 | `grok-4.5` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
|
||||||
|
| GLM-5.2 | `glm-5.2` | same | same |
|
||||||
|
| GLM-5.1 | `glm-5.1` | same | same |
|
||||||
|
| Kimi K3 | `kimi-k3` | same | same |
|
||||||
|
| Kimi K2.7 Code | `kimi-k2.7-code` | same | same |
|
||||||
|
| Kimi K2.6 | `kimi-k2.6` | same | same |
|
||||||
|
| DeepSeek V4 Pro | `deepseek-v4-pro` | same | same |
|
||||||
|
| DeepSeek V4 Flash | `deepseek-v4-flash` | same | same |
|
||||||
|
| MiMo-V2.5 | `mimo-v2.5` | same | same |
|
||||||
|
| MiMo-V2.5-Pro | `mimo-v2.5-pro` | same | same |
|
||||||
|
| MiniMax M3 | `minimax-m3` | `https://opencode.ai/zen/go/v1/messages` | @ai-sdk/anthropic |
|
||||||
|
| MiniMax M2.7 | `minimax-m2.7` | same | same |
|
||||||
|
| MiniMax M2.5 | `minimax-m2.5` | same | same |
|
||||||
|
| Qwen3.7 Max | `qwen3.7-max` | same | same |
|
||||||
|
| Qwen3.7 Plus | `qwen3.7-plus` | same | same |
|
||||||
|
| Qwen3.6 Plus | `qwen3.6-plus` | same | same |
|
||||||
|
| Hy3 | `hy3` | `https://opencode.ai/zen/go/v1/chat/completions` | @ai-sdk/openai-compatible |
|
||||||
|
|
||||||
|
**Config format:** `opencode-go/<model-id>` (e.g. `opencode-go/kimi-k3`)
|
||||||
|
|
||||||
|
**Model list API:** `https://opencode.ai/zen/go/v1/models`
|
||||||
|
|
||||||
|
## Privacy
|
||||||
|
|
||||||
|
Zero-retention policy from providers — your data is not used for model training.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Make AI coding accessible via low cost subscription
|
||||||
|
- Provide reliable access to the best open coding models
|
||||||
|
- Curate tested and benchmarked models for coding agent use
|
||||||
|
- No lock-in (use any other provider too)
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
---
|
||||||
|
title: OpenRouter
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: medium
|
||||||
|
tags: [system, openrouter, provider]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# OpenRouter
|
||||||
|
## Purpose
|
||||||
|
Fallback model provider.
|
||||||
|
## Location
|
||||||
|
Remote service; API token store.
|
||||||
|
## Current configuration
|
||||||
|
Primary key, fallback models, rate limits.
|
||||||
|
## Current state as of 2026-07-22
|
||||||
|
OpenRouter is not configured as a standalone provider in `~/.hermes/config.yaml` `custom_providers`.
|
||||||
|
OmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.
|
||||||
|
|
||||||
|
## History
|
||||||
|
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
|
||||||
|
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
|
||||||
|
- **Inferred:** Multiple backups before model/routing changes imply OpenRouter role changed during prior config iterations
|
||||||
|
- **Unknown:** Exact date OpenRouter was replaced or partially deprecated; exact models that were available
|
||||||
|
|
||||||
|
## Connected systems
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/omniroute]]
|
||||||
|
|
||||||
|
## Known issues
|
||||||
|
- Replaced: legacy Hermes autos not persistent, prompting Hermes Agent, which blocks attribution.
|
||||||
|
|
||||||
|
## Change history
|
||||||
|
- Pre-2026-07-20: OpenRouter provisioning active in some form
|
||||||
|
- 2026-07-20: Prior model-fix backup suggests routing changes around this time
|
||||||
|
|
||||||
|
## Relevant runbooks
|
||||||
|
- [[restore-openrouter]]
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
---
|
||||||
|
title: Scheduled Tasks
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-08-10
|
||||||
|
verified_on: 2026-08-10
|
||||||
|
confidence: high
|
||||||
|
tags: [cron, scheduled, automation]
|
||||||
|
sources: [live-cron-list]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Scheduled Tasks
|
||||||
|
|
||||||
|
## Count
|
||||||
|
- **Active jobs:** 28 (verified from live `hermes cron list`, 2026-08-07)
|
||||||
|
- **Previously 25** — GitHub Trending Radar Daily removed 2026-08-02; several jobs added since.
|
||||||
|
|
||||||
|
## Job Table
|
||||||
|
|
||||||
|
| Name | Schedule | Delivery | Skills | Script | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| Facebook Birthday Monthly Scrape | `0 9 1 * *` | telegram | — | — | ⚠ thread_id not found |
|
||||||
|
| Facebook Birthday Weekly Registry Sync | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ thread_id not found |
|
||||||
|
| Daily Birthday Checker | `30 7 * * *` | telegram | — | `birthday_check.py` | no-agent |
|
||||||
|
| Daily Marketing Job Radar | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | Primary `gpt-5.6-terra`; see [[runbooks/job-radar]] |
|
||||||
|
| Perth New Restaurant & Cafe Openings (Weekly) | `0 18 * * 4` | telegram | — | — | |
|
||||||
|
| AI Newsletter Curation for Patrick (Weekly) | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | |
|
||||||
|
| Enhanced Weekend Adventure Planner | `0 8 * * 5` | telegram | — | — | |
|
||||||
|
| Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||||
|
| Political News Digest (Mon/Thu) | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | |
|
||||||
|
| Uber Eats Monthly Spend Report | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | |
|
||||||
|
| Gmail Action Digest | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||||
|
| FreshRSS Midday News Briefing | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||||
|
| Workday start reminder | `0 9 * * 1-5` | telegram | — | — | |
|
||||||
|
| Important-mail monitor | `every 30m` | telegram:1793951355 | — | — | Silent when no important mail |
|
||||||
|
| Mnemosyne Auto-Consolidate | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent |
|
||||||
|
| Mnemosyne DB Backup | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent |
|
||||||
|
| Mnemosyne Health Monitor | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception |
|
||||||
|
| Mnemosyne Watchdog | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts |
|
||||||
|
| Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% |
|
||||||
|
| CT 221 Miner Watchdog | `every 15m` | telegram:1793951355 | pve-lxc-intrusion-detection | `ct221-miner-watchdog.sh` | Added 2026-08-07 after crypto miner incident |
|
||||||
|
| Mosquito Tracker | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 |
|
||||||
|
| Weekly Review | `0 18 * * 0` | origin | weekly-review | — | |
|
||||||
|
| Evening Stand-Down Briefing | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||||
|
| Job Tracker Dashboard Refresh | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||||
|
| Wiki Audit — Nightly | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||||
|
| GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | telegram:1793951355 | — | — | |
|
||||||
|
| 🦝 Hermes Config Backup to Gitea (Daily) | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||||
|
| Review Agent Estate Phase One | once at 2026-08-08 18:33 | origin | — | — | ✅ Completed 2026-08-08. See [[current-state]]. |
|
||||||
|
|
||||||
|
## Removed Jobs
|
||||||
|
| Name | Removed | Reason |
|
||||||
|
|---|---|---|
|
||||||
|
| GitHub Trending Radar — Daily Digest | 2026-08-02 | Stopped by user. Weekly roundup still active. |
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- **Session storage failures (2026-08-10):** Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError ("session storage could not be written"). Disk at 83% (11GB free), `state.db` at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See [[current-state]] for details.
|
||||||
|
- **Wiki Audit cron model drift (2026-08-07):** Global inference config changed (`smart-route` → `deepseek-paid-smart`) and the job is unpinned, causing `RuntimeError: Skipped to prevent unintended spend`. Needs pinning: `cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>`.
|
||||||
|
- **Facebook Birthday thread_id warnings:** Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread.
|
||||||
|
- **Model repinning (2026-08-08):** 13 cron jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. Dead combos included `freeapi/minimax-m2.7`, `freeapi/auto`, `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`. Jobs now running on valid model pins.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/mnemosyne]]
|
||||||
|
- [[runbooks/job-radar]]
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
---
|
||||||
|
title: Installed Skills
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-25
|
||||||
|
verified_on: 2026-07-25
|
||||||
|
confidence: high
|
||||||
|
tags: [skills, hermes]
|
||||||
|
sources: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Installed Skills
|
||||||
|
|
||||||
|
## Inventory
|
||||||
|
Total installed: 66 skills in `~/.hermes/skills/`.
|
||||||
|
|
||||||
|
## Categorized Active Skills
|
||||||
|
|
||||||
|
### Core Hermes
|
||||||
|
- `hermes` — Core agent skill
|
||||||
|
|
||||||
|
### Providers & Routing
|
||||||
|
- `omniroute` — OmniRoute routing
|
||||||
|
- `omniroute-ops` — OmniRoute diagnostics and combo management
|
||||||
|
|
||||||
|
### Memory
|
||||||
|
- `chromadb` — ChromaDB vector store
|
||||||
|
- `chromadb-preflight` — ChromaDB RAG preflight
|
||||||
|
- `chromadb-skills-rag` — Skills RAG via ChromaDB
|
||||||
|
|
||||||
|
### Browser / RPA
|
||||||
|
- `computer-use` — Desktop GUI automation
|
||||||
|
|
||||||
|
### DevOps / Homelab
|
||||||
|
- `devops` — Umbrella for service operations
|
||||||
|
- `devops-homelab-architecture-live-probe` — Live homelab diagram via probes
|
||||||
|
- `devops-proxmox-operations` — Proxmox VE REST/SSH operations
|
||||||
|
- `devops-service-management` — Miscellaneous service ops
|
||||||
|
- `devops-tailscale-web-routing` — Tailscale web routing
|
||||||
|
- `cloudflare-tunnel-ops` — Cloudflare Tunnels
|
||||||
|
- `tailscale-serve-ops` — Tailscale Serve
|
||||||
|
- `tailscale-ops` — Tailscale configuration
|
||||||
|
|
||||||
|
### Messaging
|
||||||
|
- `email` — Email from terminal
|
||||||
|
- `email-himalaya` — Himalaya CLI IMAP/SMTP
|
||||||
|
|
||||||
|
### Job / Career
|
||||||
|
- `job-hunting` — Career transition coaching
|
||||||
|
- `job-hunting-assistance` — End-to-end job hunting support
|
||||||
|
- `job-search-automation` — Autonomous job board scraper
|
||||||
|
- `job-tracker-enrichment` — Notion job tracker enrichment
|
||||||
|
|
||||||
|
### Content / Research
|
||||||
|
- `tech-ai-newsletter-digest` — Newsletter processing
|
||||||
|
- `media` — YouTube transcripts, GIF search, music generation
|
||||||
|
|
||||||
|
### Productivity
|
||||||
|
- `productivity` — Document creation, presentations, spreadsheets
|
||||||
|
- `markdown-to-anywhere` — Cross-posting service
|
||||||
|
- `weekly-review` — Sunday weekly review cron
|
||||||
|
|
||||||
|
### MCP / Skills Management
|
||||||
|
- `mcp` — MCP server tooling
|
||||||
|
- `mcp-agent-integration` — Local stdio MCP server wiring
|
||||||
|
|
||||||
|
### Unverified / Not Categorized
|
||||||
|
- 40+ skills not inspected in detail during this inventory
|
||||||
|
- Full list at `~/.hermes/skills/`
|
||||||
|
|
||||||
|
## How category discovery works
|
||||||
|
|
||||||
|
Skill categories are derived from the directory layout, not from Tool Search.
|
||||||
|
The first directory below `~/.hermes/skills/` becomes the category:
|
||||||
|
|
||||||
|
- `~/.hermes/skills/devops/my-skill/SKILL.md` → category `devops`
|
||||||
|
- `~/.hermes/skills/my-skill/SKILL.md` → no category
|
||||||
|
|
||||||
|
The skill's display name and discovery description come from the YAML
|
||||||
|
frontmatter in `SKILL.md`. Directory additions are detected automatically.
|
||||||
|
In-place edits may take up to 30 seconds to appear because the discovery cache
|
||||||
|
has a 30-second lifetime.
|
||||||
|
|
||||||
|
See [[systems/tool-search]] for the separate external-tool discovery system.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[messaging-integrations]]
|
||||||
|
- [[systems/tool-search]]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
title: Terminal Backend
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-22
|
||||||
|
updated: 2026-07-22
|
||||||
|
verified_on: 2026-07-22
|
||||||
|
confidence: high
|
||||||
|
tags: [terminal, shell, backend, persistence]
|
||||||
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Terminal Backend
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Execute shell commands, scripts, and interactive sessions for Hermes.
|
||||||
|
|
||||||
|
## Current Configuration
|
||||||
|
- **Backend:** local
|
||||||
|
- **Working directory:** `/home/hermes/workspace`
|
||||||
|
- **Timeout:** 180 s
|
||||||
|
- **Home mode:** auto
|
||||||
|
- **Container CPU:** 1
|
||||||
|
- **Persistent shell:** true
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
- Shell state (venv, env vars, aliases) persists across terminal calls
|
||||||
|
- Working directory fixed to `/home/hermes/workspace`
|
||||||
|
- No containerized terminal confirmed
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- None observed
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [[persistent-shells]]
|
||||||
|
- [[headless-server]]
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
---
|
||||||
|
title: Hermes Tool Search
|
||||||
|
type: system
|
||||||
|
status: active
|
||||||
|
created: 2026-07-25
|
||||||
|
updated: 2026-07-25
|
||||||
|
review_after: 2026-10-23
|
||||||
|
verified_on: 2026-07-25
|
||||||
|
version_applies_to: Hermes Agent on CT460
|
||||||
|
confidence: high
|
||||||
|
tags: [hermes, automation, system]
|
||||||
|
sources: [live CT460 configuration and isolated CUA validation]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes Tool Search
|
||||||
|
|
||||||
|
## Why it is enabled
|
||||||
|
|
||||||
|
Hermes previously sent every external tool schema to the model on every turn.
|
||||||
|
That used a large part of the context window before the conversation began and
|
||||||
|
made compaction happen much sooner.
|
||||||
|
|
||||||
|
Native Hermes Tool Search is now always on. Core tools remain immediately
|
||||||
|
available. External tools are discovered only when they are relevant, then
|
||||||
|
described and called through the Tool Search bridge.
|
||||||
|
|
||||||
|
## Active configuration
|
||||||
|
|
||||||
|
File: `/home/hermes/.hermes/config.yaml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
tools:
|
||||||
|
tool_search:
|
||||||
|
enabled: "on"
|
||||||
|
threshold_pct: 10
|
||||||
|
search_default_limit: 5
|
||||||
|
max_search_limit: 20
|
||||||
|
```
|
||||||
|
|
||||||
|
`enabled: "on"` is the important setting: it makes Tool Search active on every
|
||||||
|
request instead of waiting for the tool schemas to cross an automatic token
|
||||||
|
threshold.
|
||||||
|
|
||||||
|
## Verified result
|
||||||
|
|
||||||
|
- 80 external tools were deferred in the production gateway.
|
||||||
|
- Approximately 24,151 schema tokens were removed from the always-present
|
||||||
|
prompt.
|
||||||
|
- 36 core or directly visible tools remained available.
|
||||||
|
- Estimated fixed baseline fell from about 52,955 to 29,182 tokens, a reduction
|
||||||
|
of roughly 45%.
|
||||||
|
- In an isolated Computer Use test, 53 tools were discovered and the
|
||||||
|
`health_report` search, description, and call all succeeded.
|
||||||
|
- Native Tool Search tests passed: 39 tests.
|
||||||
|
|
||||||
|
Tool Search can add a small discovery call when an external tool is first
|
||||||
|
needed. It should not reduce answer quality: the model retains direct access to
|
||||||
|
core tools and can search for specialized tools by name or purpose.
|
||||||
|
|
||||||
|
## New tools and new skills
|
||||||
|
|
||||||
|
New external tools are automatically included in Tool Search after their tool
|
||||||
|
provider is loaded. They do not need to be manually assigned to a category.
|
||||||
|
|
||||||
|
Skills use a separate discovery system:
|
||||||
|
|
||||||
|
- `~/.hermes/skills/devops/my-skill/SKILL.md` is categorized as `devops`.
|
||||||
|
- A skill directly under `~/.hermes/skills/` has no category.
|
||||||
|
- The skill `name` and `description` come from YAML frontmatter in `SKILL.md`.
|
||||||
|
- Directory additions are noticed automatically. An in-place edit may take up
|
||||||
|
to 30 seconds to appear because of the skill cache.
|
||||||
|
|
||||||
|
Tool Search therefore does not decide skill categories. The first directory
|
||||||
|
below `~/.hermes/skills/` does.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
The pre-change configuration is preserved at:
|
||||||
|
|
||||||
|
`/home/hermes/.hermes/backups/config.yaml.20260725-before-tool-search`
|
||||||
|
|
||||||
|
Restore that file only if Tool Search causes a confirmed regression, validate
|
||||||
|
the YAML, and restart `hermes-gateway.service`.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[systems/hermes-agent]]
|
||||||
|
- [[systems/skills-index]]
|
||||||
|
- [[concepts/tool-calling]]
|
||||||
|
- [[runbooks/gateway-resource-resilience]]
|
||||||
Reference in New Issue
Block a user