diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 81a3e04..4f1dc1d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki type: guide status: active created: 2026-08-16 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [wiki, contributing, governance, multi-agent] sources: [] @@ -47,7 +47,7 @@ Each agent owns **its own section only**. Do not edit another agent's pages with | `agents/nanobot/` | Nanobot (CT333) | | `agents/zeroclaw/` | ZeroClaw (CT333) | | `agents/qwenpaw/` | QwenPaw (CT465) | -| `agents/open-claw/` | Open-Claw (VM 403) | +| ~~`agents/open-claw/`~~ | ~~Open-Claw (VM 403)~~ — **removed 2026-10-20; re-confirmed absent 2026-11-11** | | `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source | ## What to save (and what NOT to save) @@ -100,6 +100,15 @@ After pushing, the Quartz web view updates on the next daily maintenance run (03 4. Make your first commit: your role, host, config pointers, and known limitations. 5. Keep `updated:` honest on every edit. +> **Access paths re-verified 2026-11-11.** `http://100.118.5.51:9120/` returns **200** (as does a +> real page such as `/systems/skills-index`), the unit behind it is +> `hermes-wiki-serve.service` — **active**, listening on `0.0.0.0:9120` (a **user** unit — check it +> with `systemctl --user`, never bare `systemctl`, or it reads as dead). The external route +> `https://hermes.kangaroo-eel.ts.net/wiki/` returns **200** via the Tailscale Serve path. +> `git remote -v` confirms the remote is the documented +> `http://…@gitea:3000/Anthony/agent-estate-wiki.git` — note the URL carries a credential; it is +> deliberately **not** reproduced here, use `git remote -v` to read it. + ## Related - [[agents/index]] — estate map diff --git a/agents/index.md b/agents/index.md index dccec4f..05ba8a6 100644 --- a/agents/index.md +++ b/agents/index.md @@ -3,8 +3,8 @@ title: Agent Estate type: system status: active created: 2026-08-15 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [agents, estate, overview] --- @@ -40,6 +40,14 @@ also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 50 **ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333 (`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]]. +> **Guest lists re-probed 2026-11-11 — VM 403 and VM 401 still absent, and the estate shrank a +> little more.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs **100, 101, 110, 282** +> (VM **310** has gone since the 2026-10-19 probe listed it), and `pct list` shows CTs 103, 200, +> 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460, 465, 475, 555 +> (stopped), 600 — CT **601** is gone. Live CT names: `205 ollama`, `333 clawtest`, +> `450 hermesagent`, `600 omniroute`. `m93p` was not re-probed this pass; its 2026-10-20 list +> stands. ZeroClaw is running on CT333 (PID 142, `--allow-degraded-security`). + ## Shared Infrastructure All agents share: diff --git a/agents/open-claw.md b/agents/open-claw.md index b37cdb6..34b3b9d 100644 --- a/agents/open-claw.md +++ b/agents/open-claw.md @@ -3,8 +3,8 @@ title: Open-Claw type: system status: active created: 2026-08-15 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [openclaw, open-claw, vm403, gateway, agent] sources: [skill-inspection, config-reference] @@ -36,6 +36,12 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai 1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not merely moved between hosts: it is gone from both. +- **Guest lists re-probed 2026-11-11.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs + **100, 101, 110, 282** (VM **310** is gone too — the page previously listed it), and `pct list` + shows CTs 103, 200, 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460, + 465, 475, 555 (stopped), **600** — CT **601 is gone** as well. Live CT names read + `205 ollama`, `333 clawtest`, `450 hermesagent`, `600 omniroute`. **No VM 403 on either hypervisor, + and no `open-claw` Tailscale peer** — the "NOT PRESENT" verdict holds and is now ~3 weeks stable. - **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP `192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical. Treat the whole page as a design record, not a description of a running system. diff --git a/agents/zeroclaw.md b/agents/zeroclaw.md index 037844e..db16ad6 100644 --- a/agents/zeroclaw.md +++ b/agents/zeroclaw.md @@ -3,8 +3,8 @@ title: ZeroClaw type: system status: active created: 2026-08-15 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [zeroclaw, ct333, daemon, agent] sources: [skill-inspection, config-reference] @@ -40,6 +40,13 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with > the live command line. > `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots. +> **Re-verified 2026-11-11 — running as documented.** `pct exec 333 -- pgrep -af zeroclaw` returns +> `142 /usr/local/bin/zeroclaw daemon --allow-degraded-security` (PID has rotated from 136, up +> since boot). `/usr/local/bin/zeroclaw` is the live path; `/root/.local/bin/zeroclaw` also exists. +> CT333 is named **`clawtest`** in `pct list` and `pct exec 333 -- hostname` returns `clawtest` — +> so the `clawtest` label on this page is correct, and the contrary claim on +> [[comparisons/hermes-vs-nanobot]] has been corrected. + ## Key Paths | Item | Path | diff --git a/comparisons/hermes-vs-nanobot.md b/comparisons/hermes-vs-nanobot.md index fe4abb9..3761e49 100644 --- a/comparisons/hermes-vs-nanobot.md +++ b/comparisons/hermes-vs-nanobot.md @@ -3,8 +3,8 @@ title: Hermes vs Nanobot type: comparison status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [comparison, hermes, nanobot] sources: [raw/configs/hermes-config-sanitized.txt] @@ -29,8 +29,14 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab, > > **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified > live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host -> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in -> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC. +> is not wrong, but the host is no longer single-purpose. +> +> **Correction 2026-11-11: the "CT333 is `zeroclaw` in Proxmox naming" claim was itself wrong.** +> Live `pct list` on the PVE host (`100.96.100.82`) still reports CT333's name as **`clawtest`** +> and `pct exec 333 -- hostname` returns **`clawtest`** too — the wiki's original label is correct +> on both counts. `zeroclaw` is only the **process/daemon** that runs inside CT333, alongside +> `nanobot`. Re-verified live the same day: the peer is on Tailscale at `100.122.189.81 clawtest` +> and `http://nanobot.martinwa.org` still answers **301**. ## At a glance | | Hermes (ct460) | Nanobot (ct333 "clawtest") | diff --git a/comparisons/memory-backends.md b/comparisons/memory-backends.md index b39db9f..debbd23 100644 --- a/comparisons/memory-backends.md +++ b/comparisons/memory-backends.md @@ -3,8 +3,8 @@ title: Memory Backends type: comparison status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [comparison, memory] sources: [raw/configs/hermes-config-sanitized.txt] @@ -43,6 +43,14 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati > in cron contexts, which is why scheduled jobs produce no working memories. See > [[concepts/agent-memory]] and [[systems/mnemosyne]]. +> **Re-verified 2026-11-11 — still Mnemosyne only.** `memory.provider: mnemosyne`, +> `memory_enabled: true`, `flush_min_turns: 6`, `memory_char_limit: 5000`, +> `user_char_limit: 3000`, `nudge_interval: 10`, `user_profile_enabled: true`; Mnemosyne sub-block +> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`, +> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`, +> `shared_surface_read: true`. No `openviking` key anywhere in `config.yaml`; no migration +> performed. The OpenViking NOT-adopted verdict stands. + ## Related - [[systems/mnemosyne]] - [[systems/memory-backend]] diff --git a/comparisons/model-routers.md b/comparisons/model-routers.md index 364e80f..630ec8d 100644 --- a/comparisons/model-routers.md +++ b/comparisons/model-routers.md @@ -3,8 +3,8 @@ title: Model Routers type: comparison status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [comparison, routing] sources: [raw/configs/hermes-config-sanitized.txt] @@ -51,6 +51,16 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes. > Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude > the provider is down — it is not. +> **Re-verified 2026-11-11 — the corrections recorded here all still hold.** The live `model:` block +> is OmniRoute-primary (`custom:omni-101` / `smart-route` / `http://100.78.63.20:20128`), and +> LiteLLM is the **tail** of the six-entry fallback chain (`or-inclusionai_ling-2_6-1t` via +> `custom` → `http://litellm:4000/v1`). The five `custom_providers` and their base URLs are exactly +> as listed: `litellm` `http://litellm:4000/v1`, `guacano` `http://ollama:11000/v1`, +> `Omniroute:20128` `http://omniroute:20128/v1`, `command-code` +> `https://api.commandcode.ai/provider/v1`, `omni-101` `http://100.78.63.20:20128`. OpenRouter is +> still absent from `custom_providers`. **Not re-probed this pass:** the live `/v1/models` counts +> (1869 / 6457) and their 2026-10-20 timestamps stand as recorded — treat them as unverified. + ## Detail ### LiteLLM — fallback tail + a `custom_providers` entry diff --git a/concepts/agent-memory.md b/concepts/agent-memory.md index c6d7e1d..9d117eb 100644 --- a/concepts/agent-memory.md +++ b/concepts/agent-memory.md @@ -3,8 +3,8 @@ title: Agent Memory type: concept status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [concept, memory] sources: [raw/configs/hermes-config-sanitized.txt] @@ -37,6 +37,16 @@ How an agent retains facts/context beyond a single conversation turn or session > - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that > the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]). +> **Full `memory:` block re-read 2026-11-11 — every value unchanged.** `memory_enabled: true`, +> `provider: mnemosyne`, `memory_char_limit: 5000`, `user_profile_enabled: true`, +> `user_char_limit: 3000`, `flush_min_turns: 6`, `nudge_interval: 10`; Mnemosyne sub-block +> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`, +> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`, +> `shared_surface_read: true`, `sync_roles: [user]`, `ignore_patterns: []`, +> `skip_contexts: [cron, flush, subagent, background, skill_loop]`, +> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}`. The cron-skip is still why +> scheduled jobs like this stocktake leave no Mnemosyne trace. + ## Model - Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch. - `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened. diff --git a/concepts/model-routing.md b/concepts/model-routing.md index 75b8330..deaf914 100644 --- a/concepts/model-routing.md +++ b/concepts/model-routing.md @@ -3,8 +3,8 @@ title: Model Routing type: concept status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [routing, model, inference] sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md] @@ -16,18 +16,42 @@ sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md] How Hermes selects providers, applies combos, and enforces context limits. ## Current Flow -- Main path: `custom:litellm` → `http://litellm:4000/v1` -- Default model: `freellmapi-auto` -- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **all eight auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision` — all `no-think/freeapi/auto` as of 2026-10-25 19:13) +- Main path: `custom:omni-101` / `smart-route` → `http://100.78.63.20:20128` (the OmniRoute node + `omniroute-1`), `api_mode: chat_completions` +- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **nine auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `pulse_memory`, `skills_hub`, `title_generation`, `vision` — eight of them on `no-think/freeapi/auto`, plus `pulse_memory`; see the 2026-11-11 note below) + +> **Auxiliary slots re-read 2026-11-11 — the set has grown from eight to nine.** Eight slots +> (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, +> `title_generation`, `vision`) are unchanged: `provider: custom:omni-101`, `model: +> no-think/freeapi/auto`, each with a two-entry fallback chain +> (`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, both on `custom:omni-101`). +> **New: `pulse_memory`** — `provider: nous`, `model: anthropic/claude-haiku-4.5`, and it is the +> slot behind the live 404s recorded above: nous is out of credits for that **paid** model, so +> `pulse_memory` is failing right now (3 errors on 2026-11-10 18:02 and 19:19). Also live: +> `auxiliary.free_only: true`. > **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.** > `model.default` is **`smart-route`** on provider **`custom:omni-101`** > (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` / > `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**. -> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous` -> credential** (`Invalid refresh token` since 2026-10-22). +> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`** +> credential (`Invalid refresh token` since 2026-10-22). > → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] > +> **Re-read 2026-11-11 — the model block is unchanged, but the `nous` failure signature has moved +> on.** `model.default` is still `smart-route` on `custom:omni-101` +> (`base_url: http://100.78.63.20:20128`, `api_mode: chat_completions`, +> `persist_switch_by_default: false`) and the six-entry `fallback_providers` chain is unchanged in +> membership and order (`nous` → `opencode-zen` → `custom:guacano` → `custom:litellm` → +> `kilocode` → `custom`). The **refresh-token failure is gone from the logs**: `~/.hermes/.env` +> has no `NOUS_API_KEY` and `~/.hermes/auth.json` is live, and no 401/refresh-token error appears +> in `~/.hermes/logs/errors.log`. What *does* appear (3×, most recent **2026-11-10 19:19:33**) is +> **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5` — a **paid** +> model. So `nous` is no longer dead-at-auth; it is **out of credits for paid models**. Free nous +> models (position 1 is `meituan/longcat-2.5-preview:free`) have not been observed failing. The +> slot actually hit by that paid model is **`auxiliary.pulse_memory`** — see the auxiliary note +> below, which is currently a broken slot. +> > **The five `custom_providers` and their live base URLs:** > diff --git a/concepts/persistent-shells.md b/concepts/persistent-shells.md index 1005aa8..5549d26 100644 --- a/concepts/persistent-shells.md +++ b/concepts/persistent-shells.md @@ -3,8 +3,8 @@ title: Persistent Shells type: concept status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [concept, terminal] sources: [raw/configs/hermes-config-sanitized.txt] @@ -31,6 +31,12 @@ A background shell session that survives across separate Hermes turns/requests, > The container-* keys are inert while `backend: local` — the shell is not containerised on this > box, so `container_memory`/`container_disk` describe a path that isn't taken. +> **`terminal:` block re-read 2026-11-11 — unchanged.** `backend: local`, +> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`, +> `container_memory: 5120`, `container_disk: 51200`, `container_persistent: true`, +> `lifetime_seconds: 300`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`, +> `cwd: /home/hermes/workspace/`. The container-* keys are still inert while `backend: local`. + ## Why it matters here Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this. diff --git a/concepts/tool-calling.md b/concepts/tool-calling.md index a8aac65..4311939 100644 --- a/concepts/tool-calling.md +++ b/concepts/tool-calling.md @@ -3,8 +3,8 @@ title: Tool Calling type: concept status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [concept, tool-calling] sources: [raw/configs/hermes-config-sanitized.txt] @@ -49,6 +49,11 @@ This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runboo > (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not > assume one covers the other. +> **`tool_loop_guardrails` re-verified 2026-11-11 — unchanged.** `warnings_enabled: true`, +> `hard_stop_enabled: false`, `warn_after {exact_failure: 2, same_tool_failure: 3, +> idempotent_no_progress: 2}`, `hard_stop_after {exact_failure: 5, same_tool_failure: 8, +> idempotent_no_progress: 5}`. A genuinely stuck tool loop still only ever produces a warning. + ## Related - [[concepts/persistent-shells]] - [[runbooks/troubleshoot-tool-loop]] diff --git a/decisions/keep-openrouter-as-fallback.md b/decisions/keep-openrouter-as-fallback.md index bee8d80..e82fdc7 100644 --- a/decisions/keep-openrouter-as-fallback.md +++ b/decisions/keep-openrouter-as-fallback.md @@ -3,9 +3,9 @@ title: Keep OpenRouter as fallback type: decision status: active created: 2026-07-22 -updated: 2026-10-20 +updated: 2026-11-11 review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when -verified_on: 2026-10-20 +verified_on: 2026-11-11 confidence: high tags: [routing, provider, decision] sources: [] @@ -50,6 +50,14 @@ A self-hosted router has operated reliably for at least 30 days. > bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries > that concentration risk with it. +> **Re-verified 2026-11-11 — the decision still needs the re-affirmation it was flagged for on +> 2026-10-20, and the position is unchanged.** `custom_providers` still holds exactly +> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — **no standalone +> OpenRouter provider**. OmniRoute remains the primary path (`smart-route` on +> `custom:omni-101`), so the "revisit trigger" (a self-hosted router running reliably >30 days) is +> still met and the literal reading of the decision is still unsatisfied. Nothing new to decide +> here — the page still says what needs a human call. + ## Related - [[model-routing]] - [[systems/omniroute]] diff --git a/decisions/persistent-shell-enabled.md b/decisions/persistent-shell-enabled.md index 9dde45c..aed8e85 100644 --- a/decisions/persistent-shell-enabled.md +++ b/decisions/persistent-shell-enabled.md @@ -3,9 +3,9 @@ title: Persistent shell enabled type: decision status: active created: 2026-07-22 -updated: 2026-10-20 +updated: 2026-11-11 review_after: 2026-10-20 -verified_on: 2026-10-20 +verified_on: 2026-11-11 confidence: high tags: [automation, shell, decision] sources: [] @@ -34,5 +34,11 @@ Use case becomes critical again. > is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of > work but not indefinitely. See [[concepts/persistent-shells]]. +> **Re-verified 2026-11-11 — decision still `active`, setting still `true`.** +> `terminal.persistent_shell: true`, `backend: local`, `timeout: 180`, `home_mode: auto`, +> `container_cpu: 1`, `lifetime_seconds: 300`, `cwd: /home/hermes/workspace/`. The 5-minute idle +> teardown caveat recorded on 2026-10-20 still applies: state persists across a burst of work, not +> indefinitely. + ## Related - [[persistent-shells]] diff --git a/experiments/index.md b/experiments/index.md index 9cb4122..79e2b14 100644 --- a/experiments/index.md +++ b/experiments/index.md @@ -3,8 +3,8 @@ title: Experiments type: collection status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: medium tags: [experiments] --- @@ -40,6 +40,13 @@ Non-canonical tests, tuning passes, and change attempts. > See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in > [[systems/hermes-agent]]. +> **Backups tree re-listed 2026-11-11 — no change to report.** `~/.hermes/backups/` still holds the +> same dated incident directories named on 2026-10-20 (including +> `mnemosyne-capture-fix-20260801_180249/` and `legacy-memory-slim-20260801_180702/`, both verified +> present again during this pass), plus the rolling `config/` series. The three experiment +> snapshots this page originally listed (`pre-vision-fix-20260718-195712`, `pre-model-fix`, +> `pre-camofox-tuning`) are still gone. + ## Outcomes - **Verified:** Backups exist with the above names - **Unknown:** Whether each experiment succeeded, failed, or was reverted diff --git a/incidents/2026-08-01-mnemosyne-capture-failure.md b/incidents/2026-08-01-mnemosyne-capture-failure.md index 18bbe6a..0fdb7cf 100644 --- a/incidents/2026-08-01-mnemosyne-capture-failure.md +++ b/incidents/2026-08-01-mnemosyne-capture-failure.md @@ -3,9 +3,9 @@ title: Mnemosyne Automatic Capture Failure type: incident status: active created: 2026-08-01 -updated: 2026-10-20 +updated: 2026-11-11 resolved_on: 2026-10-20 -verified_on: 2026-10-20 +verified_on: 2026-11-11 confidence: high tags: [memory, mnemosyne, hermes, incident] sources: [live-config, mnemosyne-db, hermes-cli-e2e] @@ -82,6 +82,14 @@ no Mnemosyne trace. `~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and `~/.hermes/backups/legacy-memory-slim-20260801_180702/`. +> **Resolution re-confirmed 2026-11-11.** The whole `memory.mnemosyne` block in +> `~/.hermes/config.yaml` is still natively typed — `sync_roles: [user]` (list), +> `skip_contexts: [cron, flush, subagent, background, skill_loop]` (list), +> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}` (dict), `sleep_threshold: 50` +> (int), `ignore_patterns: []` (list), `profile_isolation: true` (bool), `auto_sleep: true`, +> `default_scope: session`, `vector_type: int8`. No stringification regression. Both backup +> directories named in the repair still exist. `status: resolved` stands. + ## Related - [[systems/mnemosyne]] - [[systems/memory-backend]] diff --git a/incidents/2026-08-29-gmail-read-state.md b/incidents/2026-08-29-gmail-read-state.md index 4dc3e91..2ad0cd2 100644 --- a/incidents/2026-08-29-gmail-read-state.md +++ b/incidents/2026-08-29-gmail-read-state.md @@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)" type: incident status: active created: 2026-08-29 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [email, gmail, cron, himalaya, incident] sources: [session:default/20260829_204103_9e1087] @@ -80,6 +80,25 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on > **not** read-state errors — but the read-state exposure persists regardless, because the > newsletter fetch path still issues `message read`. > +> **Re-verified 2026-11-11 — the cron error state has cleared; the read-state root cause has not.** +> +> | Item | State on 2026-10-20 | State on 2026-11-11 | +> |---|---|---| +> | `goimapnotify.service` | active (PID 383) | **still active**, now PID **441**, same binary/conf path | +> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False`, `last_status: ok` | +> | `fetch_newsletters.py` body-fetch call | `himalaya message read --preview …` | **identical, still unfixed** — the unsafe call is still in the file | +> | `85edf6772bca` Tech & AI Newsletter Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 07:05 | +> | `08eb86f02d37` Gmail Action Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 09:18 | +> | `06a0a1c8704f` AI Newsletter Curation for Patrick | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-05 10:41 | +> +> So 74 days after the incident: **both** original mitigations that were applied on 2026-08-29 are +> now **inert** — the monitor is still paused, but `goimapnotify` is back up, so IMAP IDLE is live +> again — and the root cause (`himalaya message read` on Anthony's INBOX) is untouched. The three +> email cron jobs that were in `error` on 2026-10-20 are now green, but that says nothing about +> the read-state exposure: a job that marks emails read and *succeeds* still pollutes the mailbox. +> Per the skill's own rule (a delivered/successful run is not evidence of correct behaviour), the +> exposure is best judged from the code path, not the cron status — and the code path is unchanged. +> > **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):** > rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be > running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in diff --git a/incidents/2026-10-22-nous-portal-refresh-token-invalid.md b/incidents/2026-10-22-nous-portal-refresh-token-invalid.md index 1df77a9..cacd5d2 100644 --- a/incidents/2026-10-22-nous-portal-refresh-token-invalid.md +++ b/incidents/2026-10-22-nous-portal-refresh-token-invalid.md @@ -3,9 +3,9 @@ title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable type: incident status: open created: 2026-10-26 -updated: 2026-10-26 -verified_on: 2026-10-26 -review_after: 2026-11-09 +updated: 2026-11-11 +verified_on: 2026-11-11 +review_after: 2026-12-11 confidence: high tags: [hermes, router, memory, incident, unresolved] sources: [gateway-log, live-config, live-probe] @@ -42,6 +42,30 @@ sources: [gateway-log, live-config, live-probe] - Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`. - Until then, a failover event will attempt a dead credential silently. +> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has +> changed shape rather than closed.** +> +> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which +> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*. +> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live. +> - What appears instead is **404 `insufficient_credits_for_paid_model`** for +> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session +> `20261031_190813_95fbde`. +> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all +> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but +> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added +> afterwards and **is** the slot producing those 404s. So the masking is intact for the original +> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on +> auth. +> - `nous` is **still the first entry** in `fallback_providers` +> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing, +> so the failover risk described below is now "may burn an attempt", not "guaranteed dead". +> +> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto +> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is +> currently erroring), and/or add credits for paid nous models. The original re-authenticate item +> is **superseded** — auth works now. + ## Related - [[systems/model-providers]] - [[systems/current-profile]] diff --git a/index.md b/index.md index dd8ca52..ab97ae8 100644 --- a/index.md +++ b/index.md @@ -3,8 +3,8 @@ title: Anthony's Systems Wiki type: collection status: active created: 2026-07-22 -updated: 2026-11-10 -verified_on: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout] sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake] diff --git a/infrastructure/credentials-map.md b/infrastructure/credentials-map.md index 7357fab..7a0da47 100644 --- a/infrastructure/credentials-map.md +++ b/infrastructure/credentials-map.md @@ -3,8 +3,8 @@ title: Credentials Map type: infrastructure status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: medium tags: [credentials, security, infrastructure] sources: [] @@ -45,6 +45,14 @@ Document where credentials are stored and what service they belong to. > No `connectors:` block is configured in `config.yaml` at all. > `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store. +> **Re-verified 2026-11-11 — both 2026-10-20 corrections still hold.** `config.yaml` still has **no** +> `imap`/`smtp`/`email`/`connectors`/`gmail`/`mattermost` keys; the email credentials still live in +> `~/.hermes/.env` (mode `0600`, keys `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`, `EMAIL_SMTP_HOST`, +> `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`, `EMAIL_POLL_INTERVAL` and +> more). `tools:` still contains only `tool_search`. `discord:` and `telegram:` are top-level +> blocks, and `platforms:` still lists `mattermost` (so Mattermost remains a plugin, not a `tools:` +> entry). `~/.hermes/auth.json` exists and holds OAuth material. + ## Rules - Credentials never written to this wiki - API vault is first-stop for any key lookup diff --git a/infrastructure/domains-and-tunnels.md b/infrastructure/domains-and-tunnels.md index 1e2ad4e..334c1b1 100644 --- a/infrastructure/domains-and-tunnels.md +++ b/infrastructure/domains-and-tunnels.md @@ -3,8 +3,8 @@ title: Domains and Tunnels type: infrastructure status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [infrastructure, network, cloudflare] sources: [raw/configs/hermes-config-sanitized.txt] @@ -59,6 +59,12 @@ The catch-all remains `service: http_status:404`. The ct207 `config.yml` still c ## Full ct207 ingress list (context, not all Hermes-related) The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later. +> **Full live re-read of ct207 `/etc/cloudflared/config.yml` on 2026-11-11 — the list is unchanged +> from the 2026-10-20 diff.** All ten additions/corrections recorded above are present and no new +> hostname has appeared; the catch-all is still `service: http_status:404`. ct207's LAN address is +> `192.168.178.132`, consistent with the `ssh-webserver.martinwa.org` → `100.82.40.23` entry. +> `mcp-portal.martinwa.org` remains **absent** from the ingress list — still unknown, still flagged. + ## History - ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up. diff --git a/log.md b/log.md index 6a0db2e..cc611b6 100644 --- a/log.md +++ b/log.md @@ -2455,3 +2455,15 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted* - No secrets, keys, tokens or passwords written. - **Post-audit housekeeping (01:0x AWST).** Vault sync committed in `/home/hermes/wiki` (**`3a19147`**, both vault conventions; 20 files, then amended with the freshness report). `wiki-maintain.sh` re-run **after** the page-date corrections → **skills 299 / 67 categories** (`systems/skills-index.md`), freshness **2026-11-10: 0 stale / 127 fresh / 3 no-date of 132**, Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. Two long-stale pages re-verified and de-flagged: [[systems/push-notifications]] (bridge sha256 `59cabfcc…` unchanged; `vory-push` relay **live 01:00:34, embedded in another Hermes process** — a `pgrep` on the script name is *expected* to return nothing while healthy) and [[incidents/2026-10-18-degenerate-stream-canned-substitute]] (6 runs since 10-18, **0 failed**; newest 2026-11-09 12:00:41 `completed`). **Pushed:** `6de35a3` (maintenance) → this. +### 2026-11-11 — Stocktake: 36 stale pages re-verified; 0 stale; one materially stale fallback chain rewritten + +- **Freshness:** the 03:00 `wiki-maintain.sh` run flagged **36 stale / 93 fresh / 3 no-date of 132** — the whole 2026-10-20 cohort (runbooks, concepts, decisions, comparisons, experiments, seven systems pages, the Gmail + Mnemosyne-Capture incidents, `CONTRIBUTING.md`, `agents/index.md`). After the corrections below, the **same-day report was regenerated** (script → fix → re-run) and reads **0 stale / 129 fresh / 3 no-date of 132**. Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. +- **Skills:** `systems/skills-index.md` regenerates at **299 skills / 67 categories** — unchanged count from 2026-11-10; the only diff is the date bump. Verified the generator's arithmetic against the live tree by hand: **300 `SKILL.md` files** (excluding `.archive/`), **1 nested duplicate** (`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` duplicating the same category+name) → **299 unique (category, skill) pairs**. The count is correct; do not "fix" it to 300 or 303. +- ⚠️ **Materially stale page rewritten — [[runbooks/test-openrouter]].** The 2026-10-20 fallback chain was wrong in four of six slots. Live `hermes fallback list` (verbatim recorded on the page): primary **`smart-route` via `custom:omni-101`**, then `meituan/longcat-2.5-preview:free` (nous), `deepseek-v4.1-flash` (opencode-zen), `deepseek-v4.1-flash` (custom:guacano), `kilo-reflection-beam-free` (custom:litellm), `poolside/laguna-m.1:free` (kilocode), `or-inclusionai_ling-2_6-1t` (custom). `mimo-v2.5`, `stepfun/step-3.7-flash:free` and `qwen3.5:397b` via ollama-cloud have **left** the chain; two new entries **joined** at 3 and 4. `ollama.com` dropped out of the runbook's URL test list; `guacano`/`litellm` are noted as **internal + key-gated (401 without a key)**. → [[concepts/model-routing]], [[comparisons/model-routers]] +- ⚠️ **A false claim corrected — [[comparisons/hermes-vs-nanobot]].** The 2026-10-20 note claimed "CT333 is `zeroclaw` in the current Proxmox naming". Live `pct list` **and** `pct exec 333 -- hostname` both return **`clawtest`** — the wiki's original label was right; `zeroclaw` is only the daemon running inside it. The page now says so, dated. +- ⚠️ **Incident signature change — [[incidents/2026-10-22-nous-portal-refresh-token-invalid]].** The refresh-token failure is **gone** (zero `invalid_grant` lines in `errors.log` through 2026-11-10; `auth.json` live, no `NOUS_API_KEY` in `.env`). What replaced it: **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5`, 3× on 2026-11-10. Cause identified: a **ninth auxiliary slot** `auxiliary.pulse_memory` points back at **`nous`** with a paid model — the 2026-10-25 remediation moved the original eight slots to `custom:omni-101` but this slot was added afterwards and is the only active nous path left. `nous` is still first in `fallback_providers`, but on a *free* model not observed failing. Open item revised from "re-authenticate" to "move `pulse_memory` or add credits". +- **[[incidents/2026-08-29-gmail-read-state]] — the cron error state cleared, the root cause did not.** `goimapnotify` is **still active** (now PID 441), the Important-mail monitor (`d40085da631c`) is **still paused**, and `fetch_newsletters.py` **still calls `himalaya message read --preview`** on Anthony's INBOX — 74 days unfixed. The three email crons that read `error` on 2026-10-20 (`85edf6772bca`, `08eb86f02d37`, `06a0a1c8704f`) are now `ok` with `failure_streak: 0`, but green cron status is not evidence of safe behaviour: a job that marks mail read *and succeeds* still pollutes the mailbox. Both original mitigations are now inert. +- **Live-vs-page diffs run (per skill pattern 12a), all claims held:** ct207 `cloudflared` ingress re-read in full (four Hermes entries unchanged, `mcp-portal` still absent — [[systems/cloudflare-tunnel]], [[infrastructure/domains-and-tunnels]]); CT205 Guanaco at `v0.8.15`/`5df435e` with Ollama 200 ([[systems/guanaco]]); CT270 `occ status` 33.0.2.2 installed/not-in-maintenance ([[systems/nextcloud]]); headless stack process/port state ([[systems/headless-server]], [[runbooks/restart-browser]]); hermes-ui 5174 on both CT460 and CT450 ([[systems/hermes-ui]]); all five `hermes-*` user units active + gateway/WebUI/Camofox effective cgroup limits ([[runbooks/restart-hermes]], [[runbooks/gateway-resource-resilience]]); `tool_loop_guardrails` and the whole `terminal:`/`tools:`/`memory.mnemosyne`/`custom_providers` blocks byte-identical; `ops` profile **still deleted 2026-10-11** while `multiplex_profiles: true` and both `profile_routes` still point at it — **31 days unresolved** ([[runbooks/multiplexer-setup]]); `/home/hermes/wiki` still a local-only repo, `/home/hermes/wiki-backups/` still absent, `~/.hermes/backups/` incident dirs intact; `m93p` **not** re-probed this pass (its 2026-10-20 guest list stands as written). +- **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`. +- **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]). +- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists"). diff --git a/raw/stocktake/freshness-20261111.txt b/raw/stocktake/freshness-20261111.txt new file mode 100644 index 0000000..23200b8 --- /dev/null +++ b/raw/stocktake/freshness-20261111.txt @@ -0,0 +1,145 @@ +# Wiki freshness audit — 2026-11-11 +Threshold: >21 days since `updated` + +## STALE (0) + + +## FRESH (129) + +- [ 0d] CONTRIBUTING.md +- [ 3d] agents/hermes-chaos.md +- [ 3d] agents/hermes-production.md +- [ 0d] agents/index.md +- [ 13d] agents/nanobot.md +- [ 0d] agents/open-claw.md +- [ 7d] agents/qwenpaw.md +- [ 0d] agents/zeroclaw.md +- [ 13d] comparisons/camoufox-vs-chrome.md +- [ 0d] comparisons/hermes-vs-nanobot.md +- [ 0d] comparisons/memory-backends.md +- [ 0d] comparisons/model-routers.md +- [ 13d] comparisons/steel-vs-camoufox.md +- [ 0d] concepts/agent-memory.md +- [ 13d] concepts/browser-automation.md +- [ 0d] concepts/model-routing.md +- [ 0d] concepts/persistent-shells.md +- [ 0d] concepts/tool-calling.md +- [ 1d] current-state.md +- [ 13d] decisions/index.md +- [ 0d] decisions/keep-openrouter-as-fallback.md +- [ 0d] decisions/persistent-shell-enabled.md +- [ 13d] decisions/rollback-steel-to-camofox.md +- [ 0d] experiments/index.md +- [ 18d] incidents/2026-07-29-mnemosyne-relay.md +- [ 0d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 18d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- [ 4d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 6d] incidents/2026-08-15-skills-index-drift.md +- [ 7d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 17d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 0d] incidents/2026-08-29-gmail-read-state.md +- [ 21d] incidents/2026-09-02-sftpgo-outage.md +- [ 7d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 13d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 16d] incidents/2026-10-03-ct450-hermes-update.md +- [ 16d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 13d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 18d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 10d] incidents/2026-10-09-memory-recall-relevance.md +- [ 5d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 15d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 1d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 15d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 20d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md +- [ 17d] incidents/2026-10-22-conduit-memory-start-timeout.md +- [ 0d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md +- [ 16d] incidents/2026-10-24-conduit-call-request-refused.md +- [ 16d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md +- [ 14d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md +- [ 12d] incidents/2026-10-27-job-radar-budget-exhausted.md +- [ 12d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md +- [ 12d] incidents/2026-10-29-conduit-voice-silence.md +- [ 11d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md +- [ 12d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md +- [ 11d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md +- [ 10d] incidents/2026-10-31-asana-access-failures.md +- [ 10d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md +- [ 10d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md +- [ 9d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md +- [ 8d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md +- [ 7d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md +- [ 5d] incidents/2026-11-03-job-radar-stream-active-timeout.md +- [ 5d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md +- [ 6d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md +- [ 2d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md +- [ 6d] incidents/2026-11-05-skills-index-generator-divergence.md +- [ 4d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md +- [ 2d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md +- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md +- [ 1d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md +- [ 1d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md +- [ 1d] index.md +- [ 4d] infrastructure/backups.md +- [ 0d] infrastructure/credentials-map.md +- [ 4d] infrastructure/docker-services.md +- [ 0d] infrastructure/domains-and-tunnels.md +- [ 4d] infrastructure/hosts.md +- [ 4d] infrastructure/network-map.md +- [ 4d] infrastructure/proxmox.md +- [ 1d] infrastructure/tailscale.md +- [ 7d] queries/retained-research-answers.md +- [ 17d] reports/communities-wa-social-media-audit-2026.md +- [ 10d] reports/uber-eats/2026-07.md +- [ 10d] reports/uber-eats/2026-08.md +- [ 10d] reports/uber-eats/2026-09.md +- [ 10d] reports/uber-eats/2026-10.md +- [ 10d] reports/uber-eats/index.md +- [ 0d] runbooks/backup-wiki.md +- [ 0d] runbooks/diagnose-docker.md +- [ 0d] runbooks/gateway-resource-resilience.md +- [ 4d] runbooks/job-radar.md +- [ 0d] runbooks/multiplexer-setup.md +- [ 16d] runbooks/provider-health.md +- [ 13d] runbooks/recover-docker-service.md +- [ 0d] runbooks/restart-browser.md +- [ 0d] runbooks/restart-hermes.md +- [ 0d] runbooks/restore-openrouter.md +- [ 0d] runbooks/test-openrouter.md +- [ 0d] runbooks/troubleshoot-tool-loop.md +- [ 16d] runbooks/update-hermes-safely.md +- [ 3d] systems/browser-backend.md +- [ 0d] systems/cloudflare-tunnel.md +- [ 2d] systems/conduit.md +- [ 16d] systems/current-profile.md +- [ 0d] systems/guanaco.md +- [ 0d] systems/headless-server.md +- [ 2d] systems/hermes-agent.md +- [ 3d] systems/hermes-hq-push.md +- [ 0d] systems/hermes-ui.md +- [ 7d] systems/home-assistant.md +- [ 12d] systems/homepage-dashboard.md +- [ 7d] systems/ignis.md +- [ 16d] systems/memory-backend.md +- [ 9d] systems/messaging-integrations.md +- [ 2d] systems/mnemosyne.md +- [ 16d] systems/model-providers.md +- [ 7d] systems/nanobot.md +- [ 0d] systems/nextcloud.md +- [ 0d] systems/obsidian.md +- [ 1d] systems/omniroute.md +- [ 3d] systems/opencode-go.md +- [ 0d] systems/openrouter.md +- [ 7d] systems/paseo.md +- [ 1d] systems/push-notifications.md +- [ 1d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 0d] systems/terminal-backend.md +- [ 0d] systems/tool-search.md + +## NO updated DATE (3) + +- SCHEMA.md +- experiments/active/example.md +- log.md + +TOTAL: 0 stale, 129 fresh, 3 no-date of 132 pages \ No newline at end of file diff --git a/runbooks/backup-wiki.md b/runbooks/backup-wiki.md index ae239ea..2b58fd2 100644 --- a/runbooks/backup-wiki.md +++ b/runbooks/backup-wiki.md @@ -3,8 +3,8 @@ title: Backup Wiki type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 last_tested: 2026-10-20 confidence: high tags: [runbook, wiki, backup] @@ -35,6 +35,12 @@ Create a durable backup of the wiki so recoverable copies exist before mutating > > Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run. +> **Re-verified 2026-11-11 — the page's own caveats still stand.** `/home/hermes/wiki-backups/` +> **still does not exist**, so this runbook still has never been run end to end. `/home/hermes/wiki` +> is still a git repo with **no remote** (`git remote -v` empty), and +> `/home/hermes/workspace/agent-estate-wiki` remains the separate authoring repo pushed to Gitea +> `Anthony/agent-estate-wiki`. Both trees are real and still need two separate backups. + ## Procedure ### 1. Sanity-check repo state diff --git a/runbooks/diagnose-docker.md b/runbooks/diagnose-docker.md index bcf5a4f..c4d228d 100644 --- a/runbooks/diagnose-docker.md +++ b/runbooks/diagnose-docker.md @@ -3,8 +3,8 @@ title: Diagnose Failed Docker Service type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 last_tested: 2026-10-20 confidence: medium tags: [runbook, docker, homelab] @@ -35,6 +35,12 @@ Diagnose container or compose issues on hosts where Docker is present. > Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** — > the main dashboard container is gone — which is exactly the symptom this runbook would be used for. +> **Scoping note re-confirmed 2026-11-11.** `id -nG hermes` still contains **no `docker` group** +> membership, so on CT460 `docker ps` as `hermes` still fails with +> `permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`. +> The rule stands: on this box use `sudo docker …`, or SSH/`pct exec` to the host that owns the +> container. (No container claims on this page were re-verified in this pass.) + ## Procedure ### 1. Check Docker daemon state diff --git a/runbooks/gateway-resource-resilience.md b/runbooks/gateway-resource-resilience.md index 14ff18d..a97c065 100644 --- a/runbooks/gateway-resource-resilience.md +++ b/runbooks/gateway-resource-resilience.md @@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience type: runbook status: active created: 2026-07-25 -updated: 2026-10-20 +updated: 2026-11-11 review_after: 2026-11-19 -verified_on: 2026-10-20 +verified_on: 2026-11-11 version_applies_to: Proxmox CT460 confidence: high tags: [hermes, infrastructure, runbook] @@ -96,6 +96,15 @@ consuming most of the container and taking Telegram down with them. > `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via > `systemctl show`, which is what the health-check section below uses. +> **All effective values re-confirmed live 2026-11-11 via `systemctl show`.** +> Gateway (`hermes-gateway.service`, active): `OOMPolicy=continue`, `MemoryLow=536870912`, +> `MemoryHigh=infinity`, `MemoryMax=infinity`, `CPUWeight=1000`, `Restart=always`; the drop-in file +> still reads `OOMPolicy=continue` / `MemoryLow=512M` / `CPUWeight=1000`. Camofox +> (`camofox-browser.service`, active): `OOMPolicy=kill`, `MemoryHigh=2147483648`, +> `MemoryMax=3221225472`, `MemorySwapMax=1073741824` — the raised 2G/3G/1G ceilings hold. Web UI +> (`hermes-webui.service`, active): `OOMPolicy=kill`, `MemoryHigh=1073741824`, +> `MemoryMax=1610612736`, `MemorySwapMax=1073741824`. Unchanged since 2026-10-20. + ## I/O priority limitation Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation diff --git a/runbooks/multiplexer-setup.md b/runbooks/multiplexer-setup.md index 009ea8b..9c937a1 100644 --- a/runbooks/multiplexer-setup.md +++ b/runbooks/multiplexer-setup.md @@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway) type: runbook status: active created: 2026-07-28 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [hermes, multiplexer, telegram, profiles, gateway] --- @@ -89,6 +89,14 @@ config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because tha Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway actually resolves them before relying on this runbook. +> **Re-probed 2026-11-11 — the situation is unchanged and the two open decisions are still open.** +> /home/hermes/.hermes/profiles/` contains **only** `.deleted/ops` (8-byte marker reading +> `deleted`, dated 2026-10-11); `~/.hermes/profiles/ops/` still does not exist. The default +> profile's config **still** carries `multiplex_profiles: true` and both `profile_routes` entries +> (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`, both `profile: ops`), and +> `telegram.allowed_chats` / `group_allowed_chats` still list all three chat IDs. So the two ops +> chats still route to a profile that no longer exists — **31 days unresolved**. + ## The Silent-Drop Pitfall The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped. diff --git a/runbooks/restart-browser.md b/runbooks/restart-browser.md index 9c76460..90c1ebe 100644 --- a/runbooks/restart-browser.md +++ b/runbooks/restart-browser.md @@ -3,8 +3,8 @@ title: Restart Browser Helper type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 last_tested: 2026-10-20 confidence: high tags: [runbook, browser, cua-driver, camofox] @@ -62,6 +62,13 @@ pkill -f "fluxbox" /home/hermes/.local/bin/cua-driver serve ``` +> **Procedure re-checked against live state 2026-11-11.** `Xvfb :99`, `x11vnc -display :99 +> -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and **both** +> websockify instances are running; `cua-driver serve` is up (PID 4176) and +> `/home/hermes/.local/bin/cua-driver --version` reports **0.28.2**, matching the newest entry in +> `/home/hermes/.cua-driver/packages/releases/` (`0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`, `0.28.2`). +> **5900 still has no listener** — the 6080 websockify still points at a dead port. All as recorded. + ## Verification - `cua-driver serve` process is present in `ps` - `5901` and `6081` are reachable diff --git a/runbooks/restart-hermes.md b/runbooks/restart-hermes.md index cf5255f..4bd0760 100644 --- a/runbooks/restart-hermes.md +++ b/runbooks/restart-hermes.md @@ -3,8 +3,8 @@ title: Restart Hermes type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 last_tested: 2026-10-20 confidence: high tags: [runbook, hermes, restart] @@ -85,6 +85,17 @@ pkill -f "hermes-webui/server.py" /home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py ``` +> **Steps re-checked live 2026-11-11.** All five user units are **active (running)**: +> `hermes-gateway.service`, `hermes-dashboard.service`, `hermes-ui.service`, +> `hermes-ui-prod.service`, `hermes-wiki-serve.service`. The gateway runs from the **uv-managed +> runtime** (`/home/hermes/.hermes/tools/python-3.14.9+202****1021-linux-x64/bin/python3 … gateway +> run`) through the `runpy` bootstrap that carries `_hermes_closure_repair()`; the dashboard is +> `hermes dashboard --host 0.0.0.0 --port 9119 --no-open`; the WebUI is still +> `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port +> 8787); `cua-driver serve` is running (PID 4176, cua-driver **0.28.2**); x11vnc is on `:99`/5901. +> Note the runtime date-stamp in the interpreter path has advanced (`…14.9+20261021…` vs the +> `…14.7+20260901…` recorded on 2026-10-20) — always resolve the path live, never hardcode it. + ## Verification ```bash hermes gateway status diff --git a/runbooks/restore-openrouter.md b/runbooks/restore-openrouter.md index b3e984b..3f2c73d 100644 --- a/runbooks/restore-openrouter.md +++ b/runbooks/restore-openrouter.md @@ -3,8 +3,8 @@ title: Restore OpenRouter connectivity type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 tags: [runbook, openrouter, recovery] sources: [] confidence: high @@ -37,6 +37,11 @@ confidence: high > > **Last successfully used: 2026-06-22** — which predates every configuration referenced on this > page. Treat the page as unverified until someone runs it end to end. +> **The "out of step" flag re-confirmed 2026-11-11.** There is still **no `GoOp` binary** (`which +> GoOp` and `/usr/local/bin/GoOp`, `/opt/GoOp` all absent), no `GoOp` subcommand in `hermes +> --help`, and **no WARP process** (`pgrep -af warp` empty) and no WARP client configured. The +> substitute procedure recorded on 2026-10-20 is therefore still the correct one. + ## Verification A test prompt returns successfully through OpenRouter. ## Escalation diff --git a/runbooks/test-openrouter.md b/runbooks/test-openrouter.md index a034514..895e490 100644 --- a/runbooks/test-openrouter.md +++ b/runbooks/test-openrouter.md @@ -3,8 +3,8 @@ title: Test OpenRouter type: runbook status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 last_tested: 2026-10-20 confidence: high tags: [runbook, openrouter, providers, fallback] @@ -24,7 +24,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar ## Prerequisites - Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the uv-managed runtime — see the note under **Last tested**) -- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000 +- Internet egress for opencode.ai, inference-api.nousresearch.com, kilo.ai (ollama.com is no + longer in the live chain — entry 3 uses the internal `guacano` endpoint instead) ## Procedure @@ -32,22 +33,38 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar ```bash hermes fallback list ``` -Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries): -1. `mimo-v2.5` via opencode-go -2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1` -3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1` -4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1` +Expected active fallbacks on this install (live `hermes fallback list`, 2026-11-11 — **6** entries): +1. `meituan/longcat-2.5-preview:free` via nous +2. `deepseek-v4.1-flash` via opencode-zen — `https://opencode.ai/zen/v1` +3. `deepseek-v4.1-flash` via custom:guacano — `http://ollama:11000/v1` +4. `kilo-reflection-beam-free` via custom:litellm — `http://litellm:4000/v1` 5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway` 6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1` -**Primary:** `meituan/longcat-2.5-preview:free` (via nous). +**Primary:** `smart-route` (via `custom:omni-101`, `http://100.78.63.20:20128`). -> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the -> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`). -> The other five entries and their order are unchanged. -> `hermes fallback list` also prints a standing notice on every invocation: -> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this -> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain. +> **Chain rewritten 2026-11-11 — the 2026-10-20 list recorded above was materially stale.** +> Live `hermes fallback list` now reads (verbatim): +> +> ``` +> Primary: smart-route (via custom:omni-101) +> Fallback chain (6 entries): +> 1. meituan/longcat-2.5-preview:free (via nous) +> 2. deepseek-v4.1-flash (via opencode-zen) [https://opencode.ai/zen/v1] +> 3. deepseek-v4.1-flash (via custom:guacano) +> 4. kilo-reflection-beam-free (via custom:litellm) +> 5. poolside/laguna-m.1:free (via kilocode) [https://api.kilo.ai/api/gateway] +> 6. or-inclusionai_ling-2_6-1t (via custom) [http://litellm:4000/v1] +> ``` +> +> What changed since the 2026-10-20 reading: the **primary moved back to OmniRoute** +> (`smart-route` via `custom:omni-101`, matching the live `model:` block — see +> [[concepts/model-routing]]); `mimo-v2.5` via opencode-go and `stepfun/step-3.7-flash:free` via +> nous have **left** the chain; `qwen3.5:397b` via ollama-cloud has **left** (so `ollama.com` no +> longer needs probing); `deepseek-v4.1-flash` via `custom:guacano` and `kilo-reflection-beam-free` +> via `custom:litellm` have **joined** at positions 3 and 4; positions 5 and 6 are unchanged. +> The standing *"Claude Subscription DirectSDK … Claude Code is not installed"* notice recorded on +> 2026-10-20 **did not print** on the 2026-11-11 run. ### 2. Test OpenRouter-style direct access if present ```bash @@ -60,7 +77,6 @@ Current install state as of last run: OpenRouter is not configured as a standalo for url in \ "https://opencode.ai/zen/v1/models" \ "https://inference-api.nousresearch.com/v1/models" \ - "https://ollama.com/v1/models" \ "https://api.kilo.ai/api/gateway/models"; do printf "%s -> " "$url" curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url" @@ -68,6 +84,13 @@ for url in \ done ``` +Entries 3 (`custom:guacano` → `http://ollama:11000/v1`) and 6 (`custom` → `http://litellm:4000/v1`) +are **internal to the estate** and both now **require authentication**: a bare `curl` returns +**401 `invalid_api_key`**. Their keys live in `~/.hermes/.env` behind each provider's `key_env` +(e.g. `HERMES_CUSTOM_LITELLM_API_KEY`). `guacano` resolves through CT205 over the tailnet/DNS name +`ollama`, so it is only reachable from inside the estate — a 401 there proves auth reached the +node, not that it is down (see [[runbooks/provider-health]]). + ### 4. Check Hermes fallback config reflects reality ```bash grep -A5 "^fallback:" ~/.hermes/config.yaml @@ -89,7 +112,8 @@ Verify provider names and URLs in config match what the user wants to use. ## Last tested 2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary -`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24), +`meituan/longcat-2.5-preview:free`), and again on **2026-11-11** against the live output recorded +above (6 entries, primary `smart-route` via `custom:omni-101`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24), local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that delegates to the uv-managed runtime at `/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the diff --git a/runbooks/troubleshoot-tool-loop.md b/runbooks/troubleshoot-tool-loop.md index 90a8ed9..4235dba 100644 --- a/runbooks/troubleshoot-tool-loop.md +++ b/runbooks/troubleshoot-tool-loop.md @@ -3,9 +3,9 @@ title: Troubleshoot Tool Loop type: runbook status: unresolved created: 2026-07-22 -updated: 2026-10-20 +updated: 2026-11-11 review_after: 2026-11-19 -verified_on: 2026-10-20 +verified_on: 2026-11-11 confidence: medium tags: [runbook, hermes] sources: [] @@ -94,6 +94,12 @@ delivered message contains real content. 2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`. +> **Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.** +> `tool_loop_guardrails`: `warnings_enabled: true`, `hard_stop_enabled: false`, +> `warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`, +> `hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`. +> `hard_stop_enabled: false` therefore still means a stuck tool loop only ever *warns*. + ## Related - [[concepts/tool-calling]] - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] diff --git a/systems/cloudflare-tunnel.md b/systems/cloudflare-tunnel.md index 4cead1e..d6d22be 100644 --- a/systems/cloudflare-tunnel.md +++ b/systems/cloudflare-tunnel.md @@ -3,8 +3,8 @@ title: Cloudflare Tunnel type: system status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [system, network, cloudflare] sources: [raw/configs/hermes-config-sanitized.txt] @@ -33,6 +33,13 @@ Exposes internal homelab services, including Hermes, to the public internet with `hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119) and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]]. +> **Ingress re-read live 2026-11-11 from `/etc/cloudflared/config.yml` on ct207 — unchanged.** +> `cloudflared.service` is still **active**, the tunnel is still `ct207-tunnel`, and the four +> Hermes-relevant entries are exactly as recorded: `hermes.martinwa.org`→`100.118.5.51:9119`, +> `hermes-chat.martinwa.org`→`100.118.5.51:8787`, `tg.hermes.martinwa.org`→`100.118.5.51:19119`, +> `hermes-desktop.martinwa.org`→`100.118.5.51:5174`. `mcp-portal.martinwa.org` is **still absent** +> from the ingress list — the 2026-10-20 "unresolved" flag stands, now three weeks old. + ## Connected systems - [[systems/hermes-agent]] — the service actually reached through this tunnel - [[infrastructure/domains-and-tunnels]] — full hostname table diff --git a/systems/guanaco.md b/systems/guanaco.md index f6922de..2656101 100644 --- a/systems/guanaco.md +++ b/systems/guanaco.md @@ -3,8 +3,8 @@ title: Guanaco type: system status: active created: 2026-08-14 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [llm, proxy, router, guanaco, ollama] sources: [live-probe, session-history] @@ -40,6 +40,13 @@ LLM proxy/router providing model routing and tool-call formatting for the Ollama ### v0.8.11 (previous) - Had known DSML tag leakage issues where malformed tool-call output could leak raw `` syntax into model context stream +> **Live probe 2026-11-11 — the v0.8.15 record holds.** `/root/.guanaco/repo` on CT205 +> (`pct exec 205`) still sits at tag **`v0.8.15`**, HEAD `5df435e3de11a51401972696d47daf9144b0d140` +> (the abbreviated `5df435e` recorded above), and the Ollama API on `localhost:11434/api/version` +> returns **200**. CT205 (`ollama`) is running; its LAN address is `192.168.178.158`. +> Note a naming nuance this page does not carry: Hermes's `guacano` provider points at +> **`http://ollama:11000/v1`**, i.e. Guanaco's own proxy port, not Ollama's `11434`. + ## What DSML Is DSML = "Daily Smart Messaging Log" — Guanaco's tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (``, bare `>` characters) into the conversation context, which confused downstream models. diff --git a/systems/headless-server.md b/systems/headless-server.md index 756462d..eb3a2a8 100644 --- a/systems/headless-server.md +++ b/systems/headless-server.md @@ -3,8 +3,8 @@ title: Headless Server type: system status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [system, browser] sources: [raw/configs/hermes-config-sanitized.txt] @@ -39,6 +39,12 @@ still binds **all interfaces** — `0.0.0.0:5901` and `[::]:5901` — while both are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`). +> **Re-verified 2026-11-11 via `ps aux` + `ss -ltnp` — the live chain is exactly as recorded.** +> One `Xvfb :99` (`-screen 0 1600x1200x24 -ac -nolisten tcp`), one `x11vnc -display :99 … +> -rfbport 5901 -nopw` bound to `0.0.0.0:5901`/`[::]:5901`, one `fluxbox`, and two `websockify` +> instances (`127.0.0.1:6080`→`5900`, `127.0.0.1:6081`→`5901`). **Port 5900 still has no listener** — +> the 6080 pair still points at a dead port. Camofox's own listener is separately live on `*:9377`. + ## Connected systems - [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]]) - [[infrastructure/hosts]] diff --git a/systems/hermes-ui.md b/systems/hermes-ui.md index a1e7900..5da1f45 100644 --- a/systems/hermes-ui.md +++ b/systems/hermes-ui.md @@ -3,8 +3,8 @@ title: hermes-ui (Browser Hermes Desktop) type: system status: live created: 2026-07-30 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [hermes, webui, desktop, browser] --- @@ -31,6 +31,12 @@ The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo > (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`, > the `/webui` route). Don't conflate 5174 / 5175 / 8787. +> **Both deployments re-verified 2026-11-11.** CT460: `0.0.0.0:5174` held by a `node` process +> (`ss -ltnp`), `/home/hermes/hermes-ui/` present. CT450: `ss -ltn` inside the container shows +> `0.0.0.0:5174` and `/home/hermes/hermes-ui` exists. The 5175/8787 distinction still holds: +> `127.0.0.1:5175` is a `python3` (the `/ui` Tailscale Serve route) and `0.0.0.0:8787` is the +> Hermes WebUI (`python`). + ## How It Works - Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default). diff --git a/systems/nextcloud.md b/systems/nextcloud.md index d5d4d76..82a0799 100644 --- a/systems/nextcloud.md +++ b/systems/nextcloud.md @@ -3,8 +3,8 @@ title: Nextcloud type: system status: active created: 2026-08-17 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [nextcloud, file-sharing, self-hosted, infrastructure] sources: [] @@ -40,6 +40,12 @@ Self-hosted file sync and sharing platform running on Proxmox LXC container. | news | 28.7.0 | active | RSS reader | | nextcloud_announcements | 5.0.0 | active | | +> **Re-verified 2026-11-11 — every fact on this page still holds.** CT270 (`nextcloud`) is running; +> `occ status` reports **installed: true, version 33.0.2.2, maintenance: false, +> needsDbUpgrade: false**; the container holds `192.168.178.56` (LAN) and `100.90.102.19` +> (Tailscale); `occ` lives at `/var/www/nextcloud/occ`. The `ncdownloader` NC33 incompatibility is +> unchanged and still the only app-level issue recorded. + ## Known Issues - **ncdownloader v1.0.23 incompatible with Nextcloud 33 (2026-08-17):** App calls `OC\Server::getAppConfig()` which was removed in NC33. Crashes the `/settings/admin/overview` page with a fatal error. Fix: disable the app via `occ app:disable ncdownloader`. No data loss — ncdownloader settings preserved in DB, app just needs a compatible update. diff --git a/systems/obsidian.md b/systems/obsidian.md index 90e4f5d..42340cd 100644 --- a/systems/obsidian.md +++ b/systems/obsidian.md @@ -3,8 +3,8 @@ title: Obsidian Vault type: systems status: active created: 2026-07-26 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [obsidian, notes, vault, homelab] --- @@ -35,6 +35,11 @@ Personal knowledge base — notes, research, projects, and agent documentation. > (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`). > Full folder table, left for reference: +> **Re-verified 2026-11-11 — the 2026-10-20 correction still holds.** `/home/hermes/obsidian-vault/` +> still contains only `50 - Research/` and `Cron Outputs/`, and is still **not** a git repo. +> `/home/hermes/wiki` is still a **local-only** git repo (`git remote -v` empty) tracking the +> `agent-estate-wiki/` mirror plus `reports/uber-eats/`. No numbered-folder vault exists on this box. + ## Folder Structure | Folder | Purpose | |---|---| diff --git a/systems/openrouter.md b/systems/openrouter.md index 659ce0c..3cafdfa 100644 --- a/systems/openrouter.md +++ b/systems/openrouter.md @@ -3,8 +3,8 @@ title: OpenRouter type: system status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: medium tags: [system, openrouter, provider] sources: [raw/configs/hermes-config-sanitized.txt] @@ -29,6 +29,10 @@ OmniRoute’s catalog includes provider names and model IDs consistent with Open > model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still > goes through OmniRoute, exactly as recorded here. +> **Re-verified 2026-11-11.** `custom_providers` in `/home/hermes/.hermes/config.yaml` still holds +> exactly five entries — `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — with +> **no `openrouter` entry**. The 2026-10-20 claim holds unchanged 22 days on. + ## History - **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths - **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent diff --git a/systems/skills-index.md b/systems/skills-index.md index 952f391..2167ec4 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-10 -verified_on: 2026-11-10 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [skills, hermes] sources: [] diff --git a/systems/terminal-backend.md b/systems/terminal-backend.md index 96e3c03..6d3af57 100644 --- a/systems/terminal-backend.md +++ b/systems/terminal-backend.md @@ -3,8 +3,8 @@ title: Terminal Backend type: system status: active created: 2026-07-22 -updated: 2026-10-20 -verified_on: 2026-10-20 +updated: 2026-11-11 +verified_on: 2026-11-11 confidence: high tags: [terminal, shell, backend, persistence] sources: [raw/configs/hermes-config-sanitized.txt] @@ -28,6 +28,13 @@ Live `terminal:` block re-read from `/home/hermes/.hermes/config.yaml` on 2026-1 `container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`. `backend: local` and `persistent_shell: true` are unchanged since 2026-07-22. +> **Full `terminal:` block re-read 2026-11-11 — every value unchanged.** `backend: local`, +> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`, +> `lifetime_seconds: 300`, `container_memory: 5120`, `container_disk: 51200`, +> `container_persistent: true`, `docker_mount_cwd_to_workspace: false`, +> `docker_run_as_host_user: true`. One nomenclature correction: the working-directory key is spelled +> **`cwd`** (`cwd: /home/hermes/workspace/`), not `workdir` — the value itself is unchanged. + ## Persistence - Shell state (venv, env vars, aliases) persists across terminal calls - Working directory fixed to `/home/hermes/workspace` diff --git a/systems/tool-search.md b/systems/tool-search.md index dfff7a6..f8f32ff 100644 --- a/systems/tool-search.md +++ b/systems/tool-search.md @@ -3,9 +3,9 @@ title: Hermes Tool Search type: system status: active created: 2026-07-25 -updated: 2026-10-20 +updated: 2026-11-11 review_after: 2026-11-20 -verified_on: 2026-10-20 +verified_on: 2026-11-11 version_applies_to: Hermes Agent on CT460 confidence: high tags: [hermes, automation, system] @@ -49,6 +49,11 @@ tools: request instead of waiting for the tool schemas to cross an automatic token threshold. +> **Config re-read 2026-11-11 — byte-for-byte unchanged.** `tools.tool_search` in +> `~/.hermes/config.yaml` still reads `enabled: "on"`, `threshold_pct: 20`, +> `search_default_limit: 8`, `max_search_limit: 50`, `listing_max_tokens: 8`. `tools:` carries no +> other key. Tool Search was exercised throughout this stocktake session. + ## Verified result - 80 external tools were deferred in the production gateway.