stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13) - skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299) - runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated - comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest - incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised - incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused) - 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists) - log.md addendum + index.md date bump; no page deleted, no secrets written
This commit is contained in:
+12
-3
@@ -3,8 +3,8 @@ title: Contributing to the Agent Estate Wiki
|
|||||||
type: guide
|
type: guide
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-16
|
created: 2026-08-16
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [wiki, contributing, governance, multi-agent]
|
tags: [wiki, contributing, governance, multi-agent]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -47,7 +47,7 @@ Each agent owns **its own section only**. Do not edit another agent's pages with
|
|||||||
| `agents/nanobot/` | Nanobot (CT333) |
|
| `agents/nanobot/` | Nanobot (CT333) |
|
||||||
| `agents/zeroclaw/` | ZeroClaw (CT333) |
|
| `agents/zeroclaw/` | ZeroClaw (CT333) |
|
||||||
| `agents/qwenpaw/` | QwenPaw (CT465) |
|
| `agents/qwenpaw/` | QwenPaw (CT465) |
|
||||||
| `agents/open-claw/` | Open-Claw (VM 403) |
|
| ~~`agents/open-claw/`~~ | ~~Open-Claw (VM 403)~~ — **removed 2026-10-20; re-confirmed absent 2026-11-11** |
|
||||||
| `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source |
|
| `shared/`, `infrastructure/`, `runbooks/`, `decisions/`, `incidents/` | Anyone, but **facts must be verified against live state** and linked to a source |
|
||||||
|
|
||||||
## What to save (and what NOT to save)
|
## What to save (and what NOT to save)
|
||||||
@@ -100,6 +100,15 @@ After pushing, the Quartz web view updates on the next daily maintenance run (03
|
|||||||
4. Make your first commit: your role, host, config pointers, and known limitations.
|
4. Make your first commit: your role, host, config pointers, and known limitations.
|
||||||
5. Keep `updated:` honest on every edit.
|
5. Keep `updated:` honest on every edit.
|
||||||
|
|
||||||
|
> **Access paths re-verified 2026-11-11.** `http://100.118.5.51:9120/` returns **200** (as does a
|
||||||
|
> real page such as `/systems/skills-index`), the unit behind it is
|
||||||
|
> `hermes-wiki-serve.service` — **active**, listening on `0.0.0.0:9120` (a **user** unit — check it
|
||||||
|
> with `systemctl --user`, never bare `systemctl`, or it reads as dead). The external route
|
||||||
|
> `https://hermes.kangaroo-eel.ts.net/wiki/` returns **200** via the Tailscale Serve path.
|
||||||
|
> `git remote -v` confirms the remote is the documented
|
||||||
|
> `http://…@gitea:3000/Anthony/agent-estate-wiki.git` — note the URL carries a credential; it is
|
||||||
|
> deliberately **not** reproduced here, use `git remote -v` to read it.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[agents/index]] — estate map
|
- [[agents/index]] — estate map
|
||||||
|
|||||||
+10
-2
@@ -3,8 +3,8 @@ title: Agent Estate
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [agents, estate, overview]
|
tags: [agents, estate, overview]
|
||||||
---
|
---
|
||||||
@@ -40,6 +40,14 @@ also has **no VM 403** (its LXC guests are 100, 101, 102, 104, 105, 221, 245, 50
|
|||||||
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
|
**ZeroClaw, by contrast, was re-verified live on 2026-10-20** and **is running** on CT333
|
||||||
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
|
(`/usr/local/bin/zeroclaw daemon --allow-degraded-security`, PID 136) — see [[agents/zeroclaw]].
|
||||||
|
|
||||||
|
> **Guest lists re-probed 2026-11-11 — VM 403 and VM 401 still absent, and the estate shrank a
|
||||||
|
> little more.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs **100, 101, 110, 282**
|
||||||
|
> (VM **310** has gone since the 2026-10-19 probe listed it), and `pct list` shows CTs 103, 200,
|
||||||
|
> 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460, 465, 475, 555
|
||||||
|
> (stopped), 600 — CT **601** is gone. Live CT names: `205 ollama`, `333 clawtest`,
|
||||||
|
> `450 hermesagent`, `600 omniroute`. `m93p` was not re-probed this pass; its 2026-10-20 list
|
||||||
|
> stands. ZeroClaw is running on CT333 (PID 142, `--allow-degraded-security`).
|
||||||
|
|
||||||
## Shared Infrastructure
|
## Shared Infrastructure
|
||||||
|
|
||||||
All agents share:
|
All agents share:
|
||||||
|
|||||||
+8
-2
@@ -3,8 +3,8 @@ title: Open-Claw
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [openclaw, open-claw, vm403, gateway, agent]
|
tags: [openclaw, open-claw, vm403, gateway, agent]
|
||||||
sources: [skill-inspection, config-reference]
|
sources: [skill-inspection, config-reference]
|
||||||
@@ -36,6 +36,12 @@ Autonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tai
|
|||||||
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
|
1263 `social-dashboard`, 1264 `openmausbot`, 998 `nightbridge` (stopped) — and **no VM 403 there
|
||||||
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
|
either**. The Tailscale peer list likewise contains **no `open-claw` node**. So Open-Claw is not
|
||||||
merely moved between hosts: it is gone from both.
|
merely moved between hosts: it is gone from both.
|
||||||
|
- **Guest lists re-probed 2026-11-11.** `qm list` on `pve` (`100.96.100.82`) now shows only VMs
|
||||||
|
**100, 101, 110, 282** (VM **310** is gone too — the page previously listed it), and `pct list`
|
||||||
|
shows CTs 103, 200, 201, 205, 207, 209 (stopped), 215, 255, 270, 299 (stopped), 333, 450, 460,
|
||||||
|
465, 475, 555 (stopped), **600** — CT **601 is gone** as well. Live CT names read
|
||||||
|
`205 ollama`, `333 clawtest`, `450 hermesagent`, `600 omniroute`. **No VM 403 on either hypervisor,
|
||||||
|
and no `open-claw` Tailscale peer** — the "NOT PRESENT" verdict holds and is now ~3 weeks stable.
|
||||||
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
|
- **Consequence for the rest of this page:** the Tailscale IP `100.64.172.34`, the LAN IP
|
||||||
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
|
`192.168.178.193` and the control UI `https://open-claw.kangaroo-eel.ts.net` are all historical.
|
||||||
Treat the whole page as a design record, not a description of a running system.
|
Treat the whole page as a design record, not a description of a running system.
|
||||||
|
|||||||
+9
-2
@@ -3,8 +3,8 @@ title: ZeroClaw
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-15
|
created: 2026-08-15
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [zeroclaw, ct333, daemon, agent]
|
tags: [zeroclaw, ct333, daemon, agent]
|
||||||
sources: [skill-inspection, config-reference]
|
sources: [skill-inspection, config-reference]
|
||||||
@@ -40,6 +40,13 @@ Autonomous AI daemon running on CT333. Handles independent agent operations with
|
|||||||
> the live command line.
|
> the live command line.
|
||||||
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
|
> `/root/.zeroclaw/` contains `config.toml` plus ~15 dated `config.toml.bak*` snapshots.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — running as documented.** `pct exec 333 -- pgrep -af zeroclaw` returns
|
||||||
|
> `142 /usr/local/bin/zeroclaw daemon --allow-degraded-security` (PID has rotated from 136, up
|
||||||
|
> since boot). `/usr/local/bin/zeroclaw` is the live path; `/root/.local/bin/zeroclaw` also exists.
|
||||||
|
> CT333 is named **`clawtest`** in `pct list` and `pct exec 333 -- hostname` returns `clawtest` —
|
||||||
|
> so the `clawtest` label on this page is correct, and the contrary claim on
|
||||||
|
> [[comparisons/hermes-vs-nanobot]] has been corrected.
|
||||||
|
|
||||||
## Key Paths
|
## Key Paths
|
||||||
|
|
||||||
| Item | Path |
|
| Item | Path |
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Hermes vs Nanobot
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, hermes, nanobot]
|
tags: [comparison, hermes, nanobot]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -29,8 +29,14 @@ Compare Hermes (this agent) with Nanobot, another agent process in the homelab,
|
|||||||
>
|
>
|
||||||
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
|
> **One caveat worth adding:** CT333 now runs **both** Nanobot *and* ZeroClaw (the latter verified
|
||||||
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
|
> live on 2026-10-20 — see [[agents/zeroclaw]]). The page's framing of CT333 as the Nanobot host
|
||||||
> is not wrong, but the host is no longer single-purpose. Note also that **CT333 is `zeroclaw` in
|
> is not wrong, but the host is no longer single-purpose.
|
||||||
> the current Proxmox naming** — the wiki uses the older "clawtest" label for the same LXC.
|
>
|
||||||
|
> **Correction 2026-11-11: the "CT333 is `zeroclaw` in Proxmox naming" claim was itself wrong.**
|
||||||
|
> Live `pct list` on the PVE host (`100.96.100.82`) still reports CT333's name as **`clawtest`**
|
||||||
|
> and `pct exec 333 -- hostname` returns **`clawtest`** too — the wiki's original label is correct
|
||||||
|
> on both counts. `zeroclaw` is only the **process/daemon** that runs inside CT333, alongside
|
||||||
|
> `nanobot`. Re-verified live the same day: the peer is on Tailscale at `100.122.189.81 clawtest`
|
||||||
|
> and `http://nanobot.martinwa.org` still answers **301**.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
| | Hermes (ct460) | Nanobot (ct333 "clawtest") |
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Memory Backends
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, memory]
|
tags: [comparison, memory]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -43,6 +43,14 @@ The three archived scripts (`hindsight-watchdog.py.archived`, `hindsight_migrati
|
|||||||
> in cron contexts, which is why scheduled jobs produce no working memories. See
|
> in cron contexts, which is why scheduled jobs produce no working memories. See
|
||||||
> [[concepts/agent-memory]] and [[systems/mnemosyne]].
|
> [[concepts/agent-memory]] and [[systems/mnemosyne]].
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — still Mnemosyne only.** `memory.provider: mnemosyne`,
|
||||||
|
> `memory_enabled: true`, `flush_min_turns: 6`, `memory_char_limit: 5000`,
|
||||||
|
> `user_char_limit: 3000`, `nudge_interval: 10`, `user_profile_enabled: true`; Mnemosyne sub-block
|
||||||
|
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
|
||||||
|
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
|
||||||
|
> `shared_surface_read: true`. No `openviking` key anywhere in `config.yaml`; no migration
|
||||||
|
> performed. The OpenViking NOT-adopted verdict stands.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[systems/memory-backend]]
|
- [[systems/memory-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Model Routers
|
|||||||
type: comparison
|
type: comparison
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [comparison, routing]
|
tags: [comparison, routing]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -51,6 +51,16 @@ Compare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.
|
|||||||
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
|
> Any runbook that probes these endpoints with a bare `curl` will see 401 and may wrongly conclude
|
||||||
> the provider is down — it is not.
|
> the provider is down — it is not.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — the corrections recorded here all still hold.** The live `model:` block
|
||||||
|
> is OmniRoute-primary (`custom:omni-101` / `smart-route` / `http://100.78.63.20:20128`), and
|
||||||
|
> LiteLLM is the **tail** of the six-entry fallback chain (`or-inclusionai_ling-2_6-1t` via
|
||||||
|
> `custom` → `http://litellm:4000/v1`). The five `custom_providers` and their base URLs are exactly
|
||||||
|
> as listed: `litellm` `http://litellm:4000/v1`, `guacano` `http://ollama:11000/v1`,
|
||||||
|
> `Omniroute:20128` `http://omniroute:20128/v1`, `command-code`
|
||||||
|
> `https://api.commandcode.ai/provider/v1`, `omni-101` `http://100.78.63.20:20128`. OpenRouter is
|
||||||
|
> still absent from `custom_providers`. **Not re-probed this pass:** the live `/v1/models` counts
|
||||||
|
> (1869 / 6457) and their 2026-10-20 timestamps stand as recorded — treat them as unverified.
|
||||||
|
|
||||||
## Detail
|
## Detail
|
||||||
|
|
||||||
### LiteLLM — fallback tail + a `custom_providers` entry
|
### LiteLLM — fallback tail + a `custom_providers` entry
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Agent Memory
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, memory]
|
tags: [concept, memory]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -37,6 +37,16 @@ How an agent retains facts/context beyond a single conversation turn or session
|
|||||||
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
|
> - `profile_isolation: true` — memories do not leak across Hermes profiles, which matters now that
|
||||||
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
|
> the `ops` profile has been deleted (see [[runbooks/multiplexer-setup]]).
|
||||||
|
|
||||||
|
> **Full `memory:` block re-read 2026-11-11 — every value unchanged.** `memory_enabled: true`,
|
||||||
|
> `provider: mnemosyne`, `memory_char_limit: 5000`, `user_profile_enabled: true`,
|
||||||
|
> `user_char_limit: 3000`, `flush_min_turns: 6`, `nudge_interval: 10`; Mnemosyne sub-block
|
||||||
|
> `auto_sleep: true`, `default_scope: session`, `vector_type: int8`, `sleep_threshold: 50`,
|
||||||
|
> `profile_isolation: true`, `shared_surface_path: data/shared/mnemosyne.db`,
|
||||||
|
> `shared_surface_read: true`, `sync_roles: [user]`, `ignore_patterns: []`,
|
||||||
|
> `skip_contexts: [cron, flush, subagent, background, skill_loop]`,
|
||||||
|
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}`. The cron-skip is still why
|
||||||
|
> scheduled jobs like this stocktake leave no Mnemosyne trace.
|
||||||
|
|
||||||
## Model
|
## Model
|
||||||
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
- Per-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.
|
||||||
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
- `scope: session` means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Model Routing
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [routing, model, inference]
|
tags: [routing, model, inference]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
||||||
@@ -16,18 +16,42 @@ sources: [raw/configs/hermes-config-sanitized.txt, current-profile.md]
|
|||||||
How Hermes selects providers, applies combos, and enforces context limits.
|
How Hermes selects providers, applies combos, and enforces context limits.
|
||||||
|
|
||||||
## Current Flow
|
## Current Flow
|
||||||
- Main path: `custom:litellm` → `http://litellm:4000/v1`
|
- Main path: `custom:omni-101` / `smart-route` → `http://100.78.63.20:20128` (the OmniRoute node
|
||||||
- Default model: `freellmapi-auto`
|
`omniroute-1`), `api_mode: chat_completions`
|
||||||
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **all eight auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`, `title_generation`, `vision` — all `no-think/freeapi/auto` as of 2026-10-25 19:13)
|
- OmniRoute used for the **default chat path** (`custom:omni-101` / `smart-route`, since 2026-10-24 19:06) and for **nine auxiliary slots** (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `pulse_memory`, `skills_hub`, `title_generation`, `vision` — eight of them on `no-think/freeapi/auto`, plus `pulse_memory`; see the 2026-11-11 note below)
|
||||||
|
|
||||||
|
> **Auxiliary slots re-read 2026-11-11 — the set has grown from eight to nine.** Eight slots
|
||||||
|
> (`approval`, `compression`, `curator`, `mcp`, `profile_describer`, `skills_hub`,
|
||||||
|
> `title_generation`, `vision`) are unchanged: `provider: custom:omni-101`, `model:
|
||||||
|
> no-think/freeapi/auto`, each with a two-entry fallback chain
|
||||||
|
> (`no-think/freeapi/deepseek-v4-flash` → `no-think/kg/kilo-auto/free`, both on `custom:omni-101`).
|
||||||
|
> **New: `pulse_memory`** — `provider: nous`, `model: anthropic/claude-haiku-4.5`, and it is the
|
||||||
|
> slot behind the live 404s recorded above: nous is out of credits for that **paid** model, so
|
||||||
|
> `pulse_memory` is failing right now (3 errors on 2026-11-10 18:02 and 19:19). Also live:
|
||||||
|
> `auxiliary.free_only: true`.
|
||||||
|
|
||||||
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
|
> **Live `model:` block re-read 2026-10-26 — the default has moved TWICE.**
|
||||||
> `model.default` is **`smart-route`** on provider **`custom:omni-101`**
|
> `model.default` is **`smart-route`** on provider **`custom:omni-101`**
|
||||||
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
|
> (`http://100.78.63.20:20128`), `api_mode: chat_completions`. It moved to `nous` /
|
||||||
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
|
> `meituan/longcat-2.5-preview:free` on 2026-10-20 22:01 and **back** on **2026-10-24 19:06:10**.
|
||||||
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`
|
> A six-provider top-level `fallback_providers` chain is live again, headed by the **dead `nous`**
|
||||||
> credential** (`Invalid refresh token` since 2026-10-22).
|
> credential (`Invalid refresh token` since 2026-10-22).
|
||||||
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
> → [[systems/current-profile]] · [[incidents/2026-10-22-nous-portal-refresh-token-invalid]]
|
||||||
>
|
>
|
||||||
|
> **Re-read 2026-11-11 — the model block is unchanged, but the `nous` failure signature has moved
|
||||||
|
> on.** `model.default` is still `smart-route` on `custom:omni-101`
|
||||||
|
> (`base_url: http://100.78.63.20:20128`, `api_mode: chat_completions`,
|
||||||
|
> `persist_switch_by_default: false`) and the six-entry `fallback_providers` chain is unchanged in
|
||||||
|
> membership and order (`nous` → `opencode-zen` → `custom:guacano` → `custom:litellm` →
|
||||||
|
> `kilocode` → `custom`). The **refresh-token failure is gone from the logs**: `~/.hermes/.env`
|
||||||
|
> has no `NOUS_API_KEY` and `~/.hermes/auth.json` is live, and no 401/refresh-token error appears
|
||||||
|
> in `~/.hermes/logs/errors.log`. What *does* appear (3×, most recent **2026-11-10 19:19:33**) is
|
||||||
|
> **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5` — a **paid**
|
||||||
|
> model. So `nous` is no longer dead-at-auth; it is **out of credits for paid models**. Free nous
|
||||||
|
> models (position 1 is `meituan/longcat-2.5-preview:free`) have not been observed failing. The
|
||||||
|
> slot actually hit by that paid model is **`auxiliary.pulse_memory`** — see the auxiliary note
|
||||||
|
> below, which is currently a broken slot.
|
||||||
|
>
|
||||||
|
|
||||||
> **The five `custom_providers` and their live base URLs:**
|
> **The five `custom_providers` and their live base URLs:**
|
||||||
>
|
>
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Persistent Shells
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, terminal]
|
tags: [concept, terminal]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -31,6 +31,12 @@ A background shell session that survives across separate Hermes turns/requests,
|
|||||||
> The container-* keys are inert while `backend: local` — the shell is not containerised on this
|
> The container-* keys are inert while `backend: local` — the shell is not containerised on this
|
||||||
> box, so `container_memory`/`container_disk` describe a path that isn't taken.
|
> box, so `container_memory`/`container_disk` describe a path that isn't taken.
|
||||||
|
|
||||||
|
> **`terminal:` block re-read 2026-11-11 — unchanged.** `backend: local`,
|
||||||
|
> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`,
|
||||||
|
> `container_memory: 5120`, `container_disk: 51200`, `container_persistent: true`,
|
||||||
|
> `lifetime_seconds: 300`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`,
|
||||||
|
> `cwd: /home/hermes/workspace/`. The container-* keys are still inert while `backend: local`.
|
||||||
|
|
||||||
## Why it matters here
|
## Why it matters here
|
||||||
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
Enables multi-step CLI workflows (e.g. a long `git` operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the *next* turn — see [[runbooks/troubleshoot-tool-loop]] for the related tool-loop guardrails that catch some of this.
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Tool Calling
|
|||||||
type: concept
|
type: concept
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [concept, tool-calling]
|
tags: [concept, tool-calling]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -49,6 +49,11 @@ This is the mechanism behind [[runbooks/troubleshoot-tool-loop]] — that runboo
|
|||||||
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
|
> (which watches *tool* calls) never fired. Two separate loop-detection mechanisms exist; do not
|
||||||
> assume one covers the other.
|
> assume one covers the other.
|
||||||
|
|
||||||
|
> **`tool_loop_guardrails` re-verified 2026-11-11 — unchanged.** `warnings_enabled: true`,
|
||||||
|
> `hard_stop_enabled: false`, `warn_after {exact_failure: 2, same_tool_failure: 3,
|
||||||
|
> idempotent_no_progress: 2}`, `hard_stop_after {exact_failure: 5, same_tool_failure: 8,
|
||||||
|
> idempotent_no_progress: 5}`. A genuinely stuck tool loop still only ever produces a warning.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[concepts/persistent-shells]]
|
- [[concepts/persistent-shells]]
|
||||||
- [[runbooks/troubleshoot-tool-loop]]
|
- [[runbooks/troubleshoot-tool-loop]]
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Keep OpenRouter as fallback
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
review_after: 2026-10-20 # was 0001-01-01 (bug), fixed during 2026-07-22 audit; real trigger is conditional, see Revisit when
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [routing, provider, decision]
|
tags: [routing, provider, decision]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -50,6 +50,14 @@ A self-hosted router has operated reliably for at least 30 days.
|
|||||||
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
|
> bursts, and the 2026-10-18 degenerate-stream delivery. A single-node fallback posture carries
|
||||||
> that concentration risk with it.
|
> that concentration risk with it.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — the decision still needs the re-affirmation it was flagged for on
|
||||||
|
> 2026-10-20, and the position is unchanged.** `custom_providers` still holds exactly
|
||||||
|
> `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — **no standalone
|
||||||
|
> OpenRouter provider**. OmniRoute remains the primary path (`smart-route` on
|
||||||
|
> `custom:omni-101`), so the "revisit trigger" (a self-hosted router running reliably >30 days) is
|
||||||
|
> still met and the literal reading of the decision is still unsatisfied. Nothing new to decide
|
||||||
|
> here — the page still says what needs a human call.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[model-routing]]
|
- [[model-routing]]
|
||||||
- [[systems/omniroute]]
|
- [[systems/omniroute]]
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Persistent shell enabled
|
|||||||
type: decision
|
type: decision
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
review_after: 2026-10-20
|
review_after: 2026-10-20
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [automation, shell, decision]
|
tags: [automation, shell, decision]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -34,5 +34,11 @@ Use case becomes critical again.
|
|||||||
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
|
> is torn down after 5 minutes idle, so "state persists across turns" is true within a burst of
|
||||||
> work but not indefinitely. See [[concepts/persistent-shells]].
|
> work but not indefinitely. See [[concepts/persistent-shells]].
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — decision still `active`, setting still `true`.**
|
||||||
|
> `terminal.persistent_shell: true`, `backend: local`, `timeout: 180`, `home_mode: auto`,
|
||||||
|
> `container_cpu: 1`, `lifetime_seconds: 300`, `cwd: /home/hermes/workspace/`. The 5-minute idle
|
||||||
|
> teardown caveat recorded on 2026-10-20 still applies: state persists across a burst of work, not
|
||||||
|
> indefinitely.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[persistent-shells]]
|
- [[persistent-shells]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Experiments
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [experiments]
|
tags: [experiments]
|
||||||
---
|
---
|
||||||
@@ -40,6 +40,13 @@ Non-canonical tests, tuning passes, and change attempts.
|
|||||||
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
|
> See [[incidents/2026-09-28-hermes-managed-runtime-deps]] and the runtime-deps notes in
|
||||||
> [[systems/hermes-agent]].
|
> [[systems/hermes-agent]].
|
||||||
|
|
||||||
|
> **Backups tree re-listed 2026-11-11 — no change to report.** `~/.hermes/backups/` still holds the
|
||||||
|
> same dated incident directories named on 2026-10-20 (including
|
||||||
|
> `mnemosyne-capture-fix-20260801_180249/` and `legacy-memory-slim-20260801_180702/`, both verified
|
||||||
|
> present again during this pass), plus the rolling `config/` series. The three experiment
|
||||||
|
> snapshots this page originally listed (`pre-vision-fix-20260718-195712`, `pre-model-fix`,
|
||||||
|
> `pre-camofox-tuning`) are still gone.
|
||||||
|
|
||||||
## Outcomes
|
## Outcomes
|
||||||
- **Verified:** Backups exist with the above names
|
- **Verified:** Backups exist with the above names
|
||||||
- **Unknown:** Whether each experiment succeeded, failed, or was reverted
|
- **Unknown:** Whether each experiment succeeded, failed, or was reverted
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Mnemosyne Automatic Capture Failure
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-01
|
created: 2026-08-01
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
resolved_on: 2026-10-20
|
resolved_on: 2026-10-20
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [memory, mnemosyne, hermes, incident]
|
tags: [memory, mnemosyne, hermes, incident]
|
||||||
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
sources: [live-config, mnemosyne-db, hermes-cli-e2e]
|
||||||
@@ -82,6 +82,14 @@ no Mnemosyne trace.
|
|||||||
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
|
`~/.hermes/backups/mnemosyne-capture-fix-20260801_180249/` and
|
||||||
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
`~/.hermes/backups/legacy-memory-slim-20260801_180702/`.
|
||||||
|
|
||||||
|
> **Resolution re-confirmed 2026-11-11.** The whole `memory.mnemosyne` block in
|
||||||
|
> `~/.hermes/config.yaml` is still natively typed — `sync_roles: [user]` (list),
|
||||||
|
> `skip_contexts: [cron, flush, subagent, background, skill_loop]` (list),
|
||||||
|
> `reflect: {disabled_for_cron: true, max_calls_per_session: 3}` (dict), `sleep_threshold: 50`
|
||||||
|
> (int), `ignore_patterns: []` (list), `profile_isolation: true` (bool), `auto_sleep: true`,
|
||||||
|
> `default_scope: session`, `vector_type: int8`. No stringification regression. Both backup
|
||||||
|
> directories named in the repair still exist. `status: resolved` stands.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/mnemosyne]]
|
- [[systems/mnemosyne]]
|
||||||
- [[systems/memory-backend]]
|
- [[systems/memory-backend]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: "Gmail Read-State Incident (2026-08-29)"
|
|||||||
type: incident
|
type: incident
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-29
|
created: 2026-08-29
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [email, gmail, cron, himalaya, incident]
|
tags: [email, gmail, cron, himalaya, incident]
|
||||||
sources: [session:default/20260829_204103_9e1087]
|
sources: [session:default/20260829_204103_9e1087]
|
||||||
@@ -80,6 +80,25 @@ Despite using `--preview`, Himalaya may still mark messages as read depending on
|
|||||||
> **not** read-state errors — but the read-state exposure persists regardless, because the
|
> **not** read-state errors — but the read-state exposure persists regardless, because the
|
||||||
> newsletter fetch path still issues `message read`.
|
> newsletter fetch path still issues `message read`.
|
||||||
>
|
>
|
||||||
|
> **Re-verified 2026-11-11 — the cron error state has cleared; the read-state root cause has not.**
|
||||||
|
>
|
||||||
|
> | Item | State on 2026-10-20 | State on 2026-11-11 |
|
||||||
|
> |---|---|---|
|
||||||
|
> | `goimapnotify.service` | active (PID 383) | **still active**, now PID **441**, same binary/conf path |
|
||||||
|
> | Important-mail monitor (`d40085da631c`) | paused | **still paused** — `enabled: False`, `last_status: ok` |
|
||||||
|
> | `fetch_newsletters.py` body-fetch call | `himalaya message read --preview …` | **identical, still unfixed** — the unsafe call is still in the file |
|
||||||
|
> | `85edf6772bca` Tech & AI Newsletter Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 07:05 |
|
||||||
|
> | `08eb86f02d37` Gmail Action Digest | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-10 09:18 |
|
||||||
|
> | `06a0a1c8704f` AI Newsletter Curation for Patrick | `error` | **`ok`**, `failure_streak: 0`, last run 2026-11-05 10:41 |
|
||||||
|
>
|
||||||
|
> So 74 days after the incident: **both** original mitigations that were applied on 2026-08-29 are
|
||||||
|
> now **inert** — the monitor is still paused, but `goimapnotify` is back up, so IMAP IDLE is live
|
||||||
|
> again — and the root cause (`himalaya message read` on Anthony's INBOX) is untouched. The three
|
||||||
|
> email cron jobs that were in `error` on 2026-10-20 are now green, but that says nothing about
|
||||||
|
> the read-state exposure: a job that marks emails read and *succeeds* still pollutes the mailbox.
|
||||||
|
> Per the skill's own rule (a delivered/successful run is not evidence of correct behaviour), the
|
||||||
|
> exposure is best judged from the code path, not the cron status — and the code path is unchanged.
|
||||||
|
>
|
||||||
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
|
> **Recommended action (not taken here — this is a wiki job, and the fix touches Anthony's mail):**
|
||||||
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
|
> rewrite `fetch_newsletters.py` to use readonly IMAP, and decide whether `goimapnotify` should be
|
||||||
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
|
> running. See [[systems/scheduled-tasks]] and the Gmail IMAP lessons in
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Nous Portal Refresh Token Invalid — Context Summaries Unavailable
|
|||||||
type: incident
|
type: incident
|
||||||
status: open
|
status: open
|
||||||
created: 2026-10-26
|
created: 2026-10-26
|
||||||
updated: 2026-10-26
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-26
|
verified_on: 2026-11-11
|
||||||
review_after: 2026-11-09
|
review_after: 2026-12-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, router, memory, incident, unresolved]
|
tags: [hermes, router, memory, incident, unresolved]
|
||||||
sources: [gateway-log, live-config, live-probe]
|
sources: [gateway-log, live-config, live-probe]
|
||||||
@@ -42,6 +42,30 @@ sources: [gateway-log, live-config, live-probe]
|
|||||||
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
- Re-authenticate the Nous Portal credential, or remove `nous` from `fallback_providers`.
|
||||||
- Until then, a failover event will attempt a dead credential silently.
|
- Until then, a failover event will attempt a dead credential silently.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — the refresh-token failure is gone from the logs, and the exposure has
|
||||||
|
> changed shape rather than closed.**
|
||||||
|
>
|
||||||
|
> - **Zero `invalid_grant` / refresh-token lines** now appear in `~/.hermes/logs/errors.log` (which
|
||||||
|
> covers the window through 2026-11-10 19:19) — so the credential is no longer failing at *auth*.
|
||||||
|
> `~/.hermes/.env` carries no `NOUS_API_KEY`; `~/.hermes/auth.json` is present and live.
|
||||||
|
> - What appears instead is **404 `insufficient_credits_for_paid_model`** for
|
||||||
|
> `anthropic/claude-haiku-4.5` — 3 occurrences, most recent **2026-11-10 19:19:33**, from session
|
||||||
|
> `20261031_190813_95fbde`.
|
||||||
|
> - **A ninth auxiliary slot now points back at `nous`.** The 2026-10-25 remediation moved all
|
||||||
|
> *eight* slots off `nous` onto `custom:omni-101` / `no-think/freeapi/auto`, but
|
||||||
|
> `auxiliary.pulse_memory` (`provider: nous`, `model: anthropic/claude-haiku-4.5`) was added
|
||||||
|
> afterwards and **is** the slot producing those 404s. So the masking is intact for the original
|
||||||
|
> eight slots but a new, single-slot nous dependency exists and is **failing on credits**, not on
|
||||||
|
> auth.
|
||||||
|
> - `nous` is **still the first entry** in `fallback_providers`
|
||||||
|
> (`meituan/longcat-2.5-preview:free`) — that is a *free* model and has not been observed failing,
|
||||||
|
> so the failover risk described below is now "may burn an attempt", not "guaranteed dead".
|
||||||
|
>
|
||||||
|
> **Revised open item:** decide whether `auxiliary.pulse_memory` should be moved onto
|
||||||
|
> `custom:omni-101` like the other eight slots (it is the only active nous path left, and it is
|
||||||
|
> currently erroring), and/or add credits for paid nous models. The original re-authenticate item
|
||||||
|
> is **superseded** — auth works now.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[systems/model-providers]]
|
- [[systems/model-providers]]
|
||||||
- [[systems/current-profile]]
|
- [[systems/current-profile]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Anthony's Systems Wiki
|
|||||||
type: collection
|
type: collection
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-10
|
updated: 2026-11-11
|
||||||
verified_on: 2026-11-10
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
tags: [index, infrastructure, systems, stream-failure, ct299, skills-index, generator-divergence, secrets-at-rest, incident, fire-claim-discard, job-radar, hook-timeout]
|
||||||
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-session-store, raw/stocktake]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Credentials Map
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [credentials, security, infrastructure]
|
tags: [credentials, security, infrastructure]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -45,6 +45,14 @@ Document where credentials are stored and what service they belong to.
|
|||||||
> No `connectors:` block is configured in `config.yaml` at all.
|
> No `connectors:` block is configured in `config.yaml` at all.
|
||||||
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
|
> `~/.hermes/auth.json` also exists and holds OAuth tokens — treat as a credential store.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — both 2026-10-20 corrections still hold.** `config.yaml` still has **no**
|
||||||
|
> `imap`/`smtp`/`email`/`connectors`/`gmail`/`mattermost` keys; the email credentials still live in
|
||||||
|
> `~/.hermes/.env` (mode `0600`, keys `EMAIL_ADDRESS`, `EMAIL_IMAP_HOST`, `EMAIL_SMTP_HOST`,
|
||||||
|
> `EMAIL_ALLOWED_USERS`, `EMAIL_HOME_ADDRESS`, `EMAIL_ALLOW_ALL_USERS`, `EMAIL_POLL_INTERVAL` and
|
||||||
|
> more). `tools:` still contains only `tool_search`. `discord:` and `telegram:` are top-level
|
||||||
|
> blocks, and `platforms:` still lists `mattermost` (so Mattermost remains a plugin, not a `tools:`
|
||||||
|
> entry). `~/.hermes/auth.json` exists and holds OAuth material.
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
- Credentials never written to this wiki
|
- Credentials never written to this wiki
|
||||||
- API vault is first-stop for any key lookup
|
- API vault is first-stop for any key lookup
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Domains and Tunnels
|
|||||||
type: infrastructure
|
type: infrastructure
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [infrastructure, network, cloudflare]
|
tags: [infrastructure, network, cloudflare]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -59,6 +59,12 @@ The catch-all remains `service: http_status:404`. The ct207 `config.yml` still c
|
|||||||
## Full ct207 ingress list (context, not all Hermes-related)
|
## Full ct207 ingress list (context, not all Hermes-related)
|
||||||
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
The tunnel also fronts: `pve.martinwa.org` (Proxmox UI), `docker.martinwa.org` (Portainer), `runtipi.martinwa.org`, `homar.martinwa.org`, `nextcloud.martinwa.org`, `readlater.martinwa.org`, `velvet.martinwa.org`, `overtchat.martinwa.org`, `freellmapi.martinwa.org`, `omnideb.martinwa.org`, `newstarter.martinwa.org`, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.
|
||||||
|
|
||||||
|
> **Full live re-read of ct207 `/etc/cloudflared/config.yml` on 2026-11-11 — the list is unchanged
|
||||||
|
> from the 2026-10-20 diff.** All ten additions/corrections recorded above are present and no new
|
||||||
|
> hostname has appeared; the catch-all is still `service: http_status:404`. ct207's LAN address is
|
||||||
|
> `192.168.178.132`, consistent with the `ssh-webserver.martinwa.org` → `100.82.40.23` entry.
|
||||||
|
> `mcp-portal.martinwa.org` remains **absent** from the ingress list — still unknown, still flagged.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
- ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up.
|
- ct207's `/etc/cloudflared/` directory has ~25 dated `config.yml.bak*` files going back to 2026-06-18, including several during ct460's own setup (`config.yml.before-hermes-ct460-20260717203530`, `...before-hermes-desktop-20260718`, `...before-hermesagent-20260718085557`) — evidence the ingress list has been actively iterated on throughout Hermes's bring-up.
|
||||||
|
|
||||||
|
|||||||
@@ -2455,3 +2455,15 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
|||||||
- No secrets, keys, tokens or passwords written.
|
- No secrets, keys, tokens or passwords written.
|
||||||
- **Post-audit housekeeping (01:0x AWST).** Vault sync committed in `/home/hermes/wiki` (**`3a19147`**, both vault conventions; 20 files, then amended with the freshness report). `wiki-maintain.sh` re-run **after** the page-date corrections → **skills 299 / 67 categories** (`systems/skills-index.md`), freshness **2026-11-10: 0 stale / 127 fresh / 3 no-date of 132**, Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. Two long-stale pages re-verified and de-flagged: [[systems/push-notifications]] (bridge sha256 `59cabfcc…` unchanged; `vory-push` relay **live 01:00:34, embedded in another Hermes process** — a `pgrep` on the script name is *expected* to return nothing while healthy) and [[incidents/2026-10-18-degenerate-stream-canned-substitute]] (6 runs since 10-18, **0 failed**; newest 2026-11-09 12:00:41 `completed`). **Pushed:** `6de35a3` (maintenance) → this.
|
- **Post-audit housekeeping (01:0x AWST).** Vault sync committed in `/home/hermes/wiki` (**`3a19147`**, both vault conventions; 20 files, then amended with the freshness report). `wiki-maintain.sh` re-run **after** the page-date corrections → **skills 299 / 67 categories** (`systems/skills-index.md`), freshness **2026-11-10: 0 stale / 127 fresh / 3 no-date of 132**, Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. Two long-stale pages re-verified and de-flagged: [[systems/push-notifications]] (bridge sha256 `59cabfcc…` unchanged; `vory-push` relay **live 01:00:34, embedded in another Hermes process** — a `pgrep` on the script name is *expected* to return nothing while healthy) and [[incidents/2026-10-18-degenerate-stream-canned-substitute]] (6 runs since 10-18, **0 failed**; newest 2026-11-09 12:00:41 `completed`). **Pushed:** `6de35a3` (maintenance) → this.
|
||||||
|
|
||||||
|
### 2026-11-11 — Stocktake: 36 stale pages re-verified; 0 stale; one materially stale fallback chain rewritten
|
||||||
|
|
||||||
|
- **Freshness:** the 03:00 `wiki-maintain.sh` run flagged **36 stale / 93 fresh / 3 no-date of 132** — the whole 2026-10-20 cohort (runbooks, concepts, decisions, comparisons, experiments, seven systems pages, the Gmail + Mnemosyne-Capture incidents, `CONTRIBUTING.md`, `agents/index.md`). After the corrections below, the **same-day report was regenerated** (script → fix → re-run) and reads **0 stale / 129 fresh / 3 no-date of 132**. Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted.
|
||||||
|
- **Skills:** `systems/skills-index.md` regenerates at **299 skills / 67 categories** — unchanged count from 2026-11-10; the only diff is the date bump. Verified the generator's arithmetic against the live tree by hand: **300 `SKILL.md` files** (excluding `.archive/`), **1 nested duplicate** (`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` duplicating the same category+name) → **299 unique (category, skill) pairs**. The count is correct; do not "fix" it to 300 or 303.
|
||||||
|
- ⚠️ **Materially stale page rewritten — [[runbooks/test-openrouter]].** The 2026-10-20 fallback chain was wrong in four of six slots. Live `hermes fallback list` (verbatim recorded on the page): primary **`smart-route` via `custom:omni-101`**, then `meituan/longcat-2.5-preview:free` (nous), `deepseek-v4.1-flash` (opencode-zen), `deepseek-v4.1-flash` (custom:guacano), `kilo-reflection-beam-free` (custom:litellm), `poolside/laguna-m.1:free` (kilocode), `or-inclusionai_ling-2_6-1t` (custom). `mimo-v2.5`, `stepfun/step-3.7-flash:free` and `qwen3.5:397b` via ollama-cloud have **left** the chain; two new entries **joined** at 3 and 4. `ollama.com` dropped out of the runbook's URL test list; `guacano`/`litellm` are noted as **internal + key-gated (401 without a key)**. → [[concepts/model-routing]], [[comparisons/model-routers]]
|
||||||
|
- ⚠️ **A false claim corrected — [[comparisons/hermes-vs-nanobot]].** The 2026-10-20 note claimed "CT333 is `zeroclaw` in the current Proxmox naming". Live `pct list` **and** `pct exec 333 -- hostname` both return **`clawtest`** — the wiki's original label was right; `zeroclaw` is only the daemon running inside it. The page now says so, dated.
|
||||||
|
- ⚠️ **Incident signature change — [[incidents/2026-10-22-nous-portal-refresh-token-invalid]].** The refresh-token failure is **gone** (zero `invalid_grant` lines in `errors.log` through 2026-11-10; `auth.json` live, no `NOUS_API_KEY` in `.env`). What replaced it: **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5`, 3× on 2026-11-10. Cause identified: a **ninth auxiliary slot** `auxiliary.pulse_memory` points back at **`nous`** with a paid model — the 2026-10-25 remediation moved the original eight slots to `custom:omni-101` but this slot was added afterwards and is the only active nous path left. `nous` is still first in `fallback_providers`, but on a *free* model not observed failing. Open item revised from "re-authenticate" to "move `pulse_memory` or add credits".
|
||||||
|
- **[[incidents/2026-08-29-gmail-read-state]] — the cron error state cleared, the root cause did not.** `goimapnotify` is **still active** (now PID 441), the Important-mail monitor (`d40085da631c`) is **still paused**, and `fetch_newsletters.py` **still calls `himalaya message read --preview`** on Anthony's INBOX — 74 days unfixed. The three email crons that read `error` on 2026-10-20 (`85edf6772bca`, `08eb86f02d37`, `06a0a1c8704f`) are now `ok` with `failure_streak: 0`, but green cron status is not evidence of safe behaviour: a job that marks mail read *and succeeds* still pollutes the mailbox. Both original mitigations are now inert.
|
||||||
|
- **Live-vs-page diffs run (per skill pattern 12a), all claims held:** ct207 `cloudflared` ingress re-read in full (four Hermes entries unchanged, `mcp-portal` still absent — [[systems/cloudflare-tunnel]], [[infrastructure/domains-and-tunnels]]); CT205 Guanaco at `v0.8.15`/`5df435e` with Ollama 200 ([[systems/guanaco]]); CT270 `occ status` 33.0.2.2 installed/not-in-maintenance ([[systems/nextcloud]]); headless stack process/port state ([[systems/headless-server]], [[runbooks/restart-browser]]); hermes-ui 5174 on both CT460 and CT450 ([[systems/hermes-ui]]); all five `hermes-*` user units active + gateway/WebUI/Camofox effective cgroup limits ([[runbooks/restart-hermes]], [[runbooks/gateway-resource-resilience]]); `tool_loop_guardrails` and the whole `terminal:`/`tools:`/`memory.mnemosyne`/`custom_providers` blocks byte-identical; `ops` profile **still deleted 2026-10-11** while `multiplex_profiles: true` and both `profile_routes` still point at it — **31 days unresolved** ([[runbooks/multiplexer-setup]]); `/home/hermes/wiki` still a local-only repo, `/home/hermes/wiki-backups/` still absent, `~/.hermes/backups/` incident dirs intact; `m93p` **not** re-probed this pass (its 2026-10-20 guest list stands as written).
|
||||||
|
- **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`.
|
||||||
|
- **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]).
|
||||||
|
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists").
|
||||||
|
|||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# Wiki freshness audit — 2026-11-11
|
||||||
|
Threshold: >21 days since `updated`
|
||||||
|
|
||||||
|
## STALE (0)
|
||||||
|
|
||||||
|
|
||||||
|
## FRESH (129)
|
||||||
|
|
||||||
|
- [ 0d] CONTRIBUTING.md
|
||||||
|
- [ 3d] agents/hermes-chaos.md
|
||||||
|
- [ 3d] agents/hermes-production.md
|
||||||
|
- [ 0d] agents/index.md
|
||||||
|
- [ 13d] agents/nanobot.md
|
||||||
|
- [ 0d] agents/open-claw.md
|
||||||
|
- [ 7d] agents/qwenpaw.md
|
||||||
|
- [ 0d] agents/zeroclaw.md
|
||||||
|
- [ 13d] comparisons/camoufox-vs-chrome.md
|
||||||
|
- [ 0d] comparisons/hermes-vs-nanobot.md
|
||||||
|
- [ 0d] comparisons/memory-backends.md
|
||||||
|
- [ 0d] comparisons/model-routers.md
|
||||||
|
- [ 13d] comparisons/steel-vs-camoufox.md
|
||||||
|
- [ 0d] concepts/agent-memory.md
|
||||||
|
- [ 13d] concepts/browser-automation.md
|
||||||
|
- [ 0d] concepts/model-routing.md
|
||||||
|
- [ 0d] concepts/persistent-shells.md
|
||||||
|
- [ 0d] concepts/tool-calling.md
|
||||||
|
- [ 1d] current-state.md
|
||||||
|
- [ 13d] decisions/index.md
|
||||||
|
- [ 0d] decisions/keep-openrouter-as-fallback.md
|
||||||
|
- [ 0d] decisions/persistent-shell-enabled.md
|
||||||
|
- [ 13d] decisions/rollback-steel-to-camofox.md
|
||||||
|
- [ 0d] experiments/index.md
|
||||||
|
- [ 18d] incidents/2026-07-29-mnemosyne-relay.md
|
||||||
|
- [ 0d] incidents/2026-08-01-mnemosyne-capture-failure.md
|
||||||
|
- [ 18d] incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||||
|
- [ 4d] incidents/2026-08-08-gitea-config-backup-push-dead.md
|
||||||
|
- [ 6d] incidents/2026-08-15-skills-index-drift.md
|
||||||
|
- [ 7d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
|
||||||
|
- [ 17d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
|
||||||
|
- [ 0d] incidents/2026-08-29-gmail-read-state.md
|
||||||
|
- [ 21d] incidents/2026-09-02-sftpgo-outage.md
|
||||||
|
- [ 7d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||||
|
- [ 13d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||||
|
- [ 16d] incidents/2026-10-03-ct450-hermes-update.md
|
||||||
|
- [ 16d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
|
||||||
|
- [ 13d] incidents/2026-10-05-dead-model-pin-repin.md
|
||||||
|
- [ 18d] incidents/2026-10-09-cron-provider-timeout-storm.md
|
||||||
|
- [ 10d] incidents/2026-10-09-memory-recall-relevance.md
|
||||||
|
- [ 5d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
|
||||||
|
- [ 15d] incidents/2026-10-16-omniroute-503-resource-pressure.md
|
||||||
|
- [ 1d] incidents/2026-10-18-degenerate-stream-canned-substitute.md
|
||||||
|
- [ 15d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md
|
||||||
|
- [ 20d] incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
|
||||||
|
- [ 17d] incidents/2026-10-22-conduit-memory-start-timeout.md
|
||||||
|
- [ 0d] incidents/2026-10-22-nous-portal-refresh-token-invalid.md
|
||||||
|
- [ 16d] incidents/2026-10-24-conduit-call-request-refused.md
|
||||||
|
- [ 16d] incidents/2026-10-25-gmail-digest-runaway-and-refire.md
|
||||||
|
- [ 14d] incidents/2026-10-26-planned-restart-orphaned-cron-run.md
|
||||||
|
- [ 12d] incidents/2026-10-27-job-radar-budget-exhausted.md
|
||||||
|
- [ 12d] incidents/2026-10-28-job-radar-stage1-iteration-cap.md
|
||||||
|
- [ 12d] incidents/2026-10-29-conduit-voice-silence.md
|
||||||
|
- [ 11d] incidents/2026-10-30-ct460-host-outage-scheduler-gap.md
|
||||||
|
- [ 12d] incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md
|
||||||
|
- [ 11d] incidents/2026-10-30-mnemosyne-health-monitor-paused.md
|
||||||
|
- [ 10d] incidents/2026-10-31-asana-access-failures.md
|
||||||
|
- [ 10d] incidents/2026-10-31-conduit-voice-local-mode-not-applying.md
|
||||||
|
- [ 10d] incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md
|
||||||
|
- [ 9d] incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md
|
||||||
|
- [ 8d] incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md
|
||||||
|
- [ 7d] incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md
|
||||||
|
- [ 5d] incidents/2026-11-03-job-radar-stream-active-timeout.md
|
||||||
|
- [ 5d] incidents/2026-11-04-upstream-stream-failure-0700-cohort.md
|
||||||
|
- [ 6d] incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md
|
||||||
|
- [ 2d] incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md
|
||||||
|
- [ 6d] incidents/2026-11-05-skills-index-generator-divergence.md
|
||||||
|
- [ 4d] incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.md
|
||||||
|
- [ 2d] incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
|
||||||
|
- [ 2d] incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md
|
||||||
|
- [ 1d] incidents/2026-11-09-job-radar-stage1-stall-silent-noop.md
|
||||||
|
- [ 1d] incidents/2026-11-09-stocktake-completes-with-zero-edits.md
|
||||||
|
- [ 1d] index.md
|
||||||
|
- [ 4d] infrastructure/backups.md
|
||||||
|
- [ 0d] infrastructure/credentials-map.md
|
||||||
|
- [ 4d] infrastructure/docker-services.md
|
||||||
|
- [ 0d] infrastructure/domains-and-tunnels.md
|
||||||
|
- [ 4d] infrastructure/hosts.md
|
||||||
|
- [ 4d] infrastructure/network-map.md
|
||||||
|
- [ 4d] infrastructure/proxmox.md
|
||||||
|
- [ 1d] infrastructure/tailscale.md
|
||||||
|
- [ 7d] queries/retained-research-answers.md
|
||||||
|
- [ 17d] reports/communities-wa-social-media-audit-2026.md
|
||||||
|
- [ 10d] reports/uber-eats/2026-07.md
|
||||||
|
- [ 10d] reports/uber-eats/2026-08.md
|
||||||
|
- [ 10d] reports/uber-eats/2026-09.md
|
||||||
|
- [ 10d] reports/uber-eats/2026-10.md
|
||||||
|
- [ 10d] reports/uber-eats/index.md
|
||||||
|
- [ 0d] runbooks/backup-wiki.md
|
||||||
|
- [ 0d] runbooks/diagnose-docker.md
|
||||||
|
- [ 0d] runbooks/gateway-resource-resilience.md
|
||||||
|
- [ 4d] runbooks/job-radar.md
|
||||||
|
- [ 0d] runbooks/multiplexer-setup.md
|
||||||
|
- [ 16d] runbooks/provider-health.md
|
||||||
|
- [ 13d] runbooks/recover-docker-service.md
|
||||||
|
- [ 0d] runbooks/restart-browser.md
|
||||||
|
- [ 0d] runbooks/restart-hermes.md
|
||||||
|
- [ 0d] runbooks/restore-openrouter.md
|
||||||
|
- [ 0d] runbooks/test-openrouter.md
|
||||||
|
- [ 0d] runbooks/troubleshoot-tool-loop.md
|
||||||
|
- [ 16d] runbooks/update-hermes-safely.md
|
||||||
|
- [ 3d] systems/browser-backend.md
|
||||||
|
- [ 0d] systems/cloudflare-tunnel.md
|
||||||
|
- [ 2d] systems/conduit.md
|
||||||
|
- [ 16d] systems/current-profile.md
|
||||||
|
- [ 0d] systems/guanaco.md
|
||||||
|
- [ 0d] systems/headless-server.md
|
||||||
|
- [ 2d] systems/hermes-agent.md
|
||||||
|
- [ 3d] systems/hermes-hq-push.md
|
||||||
|
- [ 0d] systems/hermes-ui.md
|
||||||
|
- [ 7d] systems/home-assistant.md
|
||||||
|
- [ 12d] systems/homepage-dashboard.md
|
||||||
|
- [ 7d] systems/ignis.md
|
||||||
|
- [ 16d] systems/memory-backend.md
|
||||||
|
- [ 9d] systems/messaging-integrations.md
|
||||||
|
- [ 2d] systems/mnemosyne.md
|
||||||
|
- [ 16d] systems/model-providers.md
|
||||||
|
- [ 7d] systems/nanobot.md
|
||||||
|
- [ 0d] systems/nextcloud.md
|
||||||
|
- [ 0d] systems/obsidian.md
|
||||||
|
- [ 1d] systems/omniroute.md
|
||||||
|
- [ 3d] systems/opencode-go.md
|
||||||
|
- [ 0d] systems/openrouter.md
|
||||||
|
- [ 7d] systems/paseo.md
|
||||||
|
- [ 1d] systems/push-notifications.md
|
||||||
|
- [ 1d] systems/scheduled-tasks.md
|
||||||
|
- [ 0d] systems/skills-index.md
|
||||||
|
- [ 0d] systems/terminal-backend.md
|
||||||
|
- [ 0d] systems/tool-search.md
|
||||||
|
|
||||||
|
## NO updated DATE (3)
|
||||||
|
|
||||||
|
- SCHEMA.md
|
||||||
|
- experiments/active/example.md
|
||||||
|
- log.md
|
||||||
|
|
||||||
|
TOTAL: 0 stale, 129 fresh, 3 no-date of 132 pages
|
||||||
@@ -3,8 +3,8 @@ title: Backup Wiki
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
last_tested: 2026-10-20
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, wiki, backup]
|
tags: [runbook, wiki, backup]
|
||||||
@@ -35,6 +35,12 @@ Create a durable backup of the wiki so recoverable copies exist before mutating
|
|||||||
>
|
>
|
||||||
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
|
> Also note `/home/hermes/wiki-backups/` does not yet exist — this runbook has never been run.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — the page's own caveats still stand.** `/home/hermes/wiki-backups/`
|
||||||
|
> **still does not exist**, so this runbook still has never been run end to end. `/home/hermes/wiki`
|
||||||
|
> is still a git repo with **no remote** (`git remote -v` empty), and
|
||||||
|
> `/home/hermes/workspace/agent-estate-wiki` remains the separate authoring repo pushed to Gitea
|
||||||
|
> `Anthony/agent-estate-wiki`. Both trees are real and still need two separate backups.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Sanity-check repo state
|
### 1. Sanity-check repo state
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Diagnose Failed Docker Service
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
last_tested: 2026-10-20
|
last_tested: 2026-10-20
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [runbook, docker, homelab]
|
tags: [runbook, docker, homelab]
|
||||||
@@ -35,6 +35,12 @@ Diagnose container or compose issues on hosts where Docker is present.
|
|||||||
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
> Note also that `systems/homepage-dashboard.md` now records the CT201 stack as **degraded** —
|
||||||
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
> the main dashboard container is gone — which is exactly the symptom this runbook would be used for.
|
||||||
|
|
||||||
|
> **Scoping note re-confirmed 2026-11-11.** `id -nG hermes` still contains **no `docker` group**
|
||||||
|
> membership, so on CT460 `docker ps` as `hermes` still fails with
|
||||||
|
> `permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`.
|
||||||
|
> The rule stands: on this box use `sudo docker …`, or SSH/`pct exec` to the host that owns the
|
||||||
|
> container. (No container claims on this page were re-verified in this pass.)
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
### 1. Check Docker daemon state
|
### 1. Check Docker daemon state
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Hermes Gateway Resource Resilience
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-25
|
created: 2026-07-25
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
review_after: 2026-11-19
|
review_after: 2026-11-19
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
version_applies_to: Proxmox CT460
|
version_applies_to: Proxmox CT460
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, infrastructure, runbook]
|
tags: [hermes, infrastructure, runbook]
|
||||||
@@ -96,6 +96,15 @@ consuming most of the container and taking Telegram down with them.
|
|||||||
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
> `hermes` user cannot read (permission denied) — its *effective* values are still verifiable via
|
||||||
> `systemctl show`, which is what the health-check section below uses.
|
> `systemctl show`, which is what the health-check section below uses.
|
||||||
|
|
||||||
|
> **All effective values re-confirmed live 2026-11-11 via `systemctl show`.**
|
||||||
|
> Gateway (`hermes-gateway.service`, active): `OOMPolicy=continue`, `MemoryLow=536870912`,
|
||||||
|
> `MemoryHigh=infinity`, `MemoryMax=infinity`, `CPUWeight=1000`, `Restart=always`; the drop-in file
|
||||||
|
> still reads `OOMPolicy=continue` / `MemoryLow=512M` / `CPUWeight=1000`. Camofox
|
||||||
|
> (`camofox-browser.service`, active): `OOMPolicy=kill`, `MemoryHigh=2147483648`,
|
||||||
|
> `MemoryMax=3221225472`, `MemorySwapMax=1073741824` — the raised 2G/3G/1G ceilings hold. Web UI
|
||||||
|
> (`hermes-webui.service`, active): `OOMPolicy=kill`, `MemoryHigh=1073741824`,
|
||||||
|
> `MemoryMax=1610612736`, `MemorySwapMax=1073741824`. Unchanged since 2026-10-20.
|
||||||
|
|
||||||
## I/O priority limitation
|
## I/O priority limitation
|
||||||
|
|
||||||
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
Do not add `IOWeight` to the gateway drop-in unless Proxmox cgroup delegation
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Multiplexer Setup (Multi-Profile Gateway)
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-28
|
created: 2026-07-28
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
tags: [hermes, multiplexer, telegram, profiles, gateway]
|
||||||
---
|
---
|
||||||
@@ -89,6 +89,14 @@ config can no longer be read at `~/.hermes/profiles/ops/config.yaml` because tha
|
|||||||
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
|
Until that is settled, treat the ops chat IDs as **possibly unroutable** and verify the gateway
|
||||||
actually resolves them before relying on this runbook.
|
actually resolves them before relying on this runbook.
|
||||||
|
|
||||||
|
> **Re-probed 2026-11-11 — the situation is unchanged and the two open decisions are still open.**
|
||||||
|
> /home/hermes/.hermes/profiles/` contains **only** `.deleted/ops` (8-byte marker reading
|
||||||
|
> `deleted`, dated 2026-10-11); `~/.hermes/profiles/ops/` still does not exist. The default
|
||||||
|
> profile's config **still** carries `multiplex_profiles: true` and both `profile_routes` entries
|
||||||
|
> (`ops-group` → `-1003914987043`, `ops-group-2` → `-1003932503629`, both `profile: ops`), and
|
||||||
|
> `telegram.allowed_chats` / `group_allowed_chats` still list all three chat IDs. So the two ops
|
||||||
|
> chats still route to a profile that no longer exists — **31 days unresolved**.
|
||||||
|
|
||||||
## The Silent-Drop Pitfall
|
## The Silent-Drop Pitfall
|
||||||
|
|
||||||
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
The `allowed_chats` gate runs on the **default profile's adapter** BEFORE `profile_routes` stamps `source.profile`. If a chat is only authorized in the secondary profile's config but not the default's, messages from that chat are silently dropped.
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Restart Browser Helper
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
last_tested: 2026-10-20
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, browser, cua-driver, camofox]
|
tags: [runbook, browser, cua-driver, camofox]
|
||||||
@@ -62,6 +62,13 @@ pkill -f "fluxbox"
|
|||||||
/home/hermes/.local/bin/cua-driver serve
|
/home/hermes/.local/bin/cua-driver serve
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Procedure re-checked against live state 2026-11-11.** `Xvfb :99`, `x11vnc -display :99
|
||||||
|
> -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw`, `fluxbox` and **both**
|
||||||
|
> websockify instances are running; `cua-driver serve` is up (PID 4176) and
|
||||||
|
> `/home/hermes/.local/bin/cua-driver --version` reports **0.28.2**, matching the newest entry in
|
||||||
|
> `/home/hermes/.cua-driver/packages/releases/` (`0.19.3`, `0.22.2`, `0.23.2`, `0.24.0`, `0.28.2`).
|
||||||
|
> **5900 still has no listener** — the 6080 websockify still points at a dead port. All as recorded.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
- `cua-driver serve` process is present in `ps`
|
- `cua-driver serve` process is present in `ps`
|
||||||
- `5901` and `6081` are reachable
|
- `5901` and `6081` are reachable
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Restart Hermes
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
last_tested: 2026-10-20
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, hermes, restart]
|
tags: [runbook, hermes, restart]
|
||||||
@@ -85,6 +85,17 @@ pkill -f "hermes-webui/server.py"
|
|||||||
/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py
|
/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Steps re-checked live 2026-11-11.** All five user units are **active (running)**:
|
||||||
|
> `hermes-gateway.service`, `hermes-dashboard.service`, `hermes-ui.service`,
|
||||||
|
> `hermes-ui-prod.service`, `hermes-wiki-serve.service`. The gateway runs from the **uv-managed
|
||||||
|
> runtime** (`/home/hermes/.hermes/tools/python-3.14.9+202****1021-linux-x64/bin/python3 … gateway
|
||||||
|
> run`) through the `runpy` bootstrap that carries `_hermes_closure_repair()`; the dashboard is
|
||||||
|
> `hermes dashboard --host 0.0.0.0 --port 9119 --no-open`; the WebUI is still
|
||||||
|
> `/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py` (port
|
||||||
|
> 8787); `cua-driver serve` is running (PID 4176, cua-driver **0.28.2**); x11vnc is on `:99`/5901.
|
||||||
|
> Note the runtime date-stamp in the interpreter path has advanced (`…14.9+20261021…` vs the
|
||||||
|
> `…14.7+20260901…` recorded on 2026-10-20) — always resolve the path live, never hardcode it.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
```bash
|
```bash
|
||||||
hermes gateway status
|
hermes gateway status
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Restore OpenRouter connectivity
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
tags: [runbook, openrouter, recovery]
|
tags: [runbook, openrouter, recovery]
|
||||||
sources: []
|
sources: []
|
||||||
confidence: high
|
confidence: high
|
||||||
@@ -37,6 +37,11 @@ confidence: high
|
|||||||
>
|
>
|
||||||
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
|
> **Last successfully used: 2026-06-22** — which predates every configuration referenced on this
|
||||||
> page. Treat the page as unverified until someone runs it end to end.
|
> page. Treat the page as unverified until someone runs it end to end.
|
||||||
|
> **The "out of step" flag re-confirmed 2026-11-11.** There is still **no `GoOp` binary** (`which
|
||||||
|
> GoOp` and `/usr/local/bin/GoOp`, `/opt/GoOp` all absent), no `GoOp` subcommand in `hermes
|
||||||
|
> --help`, and **no WARP process** (`pgrep -af warp` empty) and no WARP client configured. The
|
||||||
|
> substitute procedure recorded on 2026-10-20 is therefore still the correct one.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
A test prompt returns successfully through OpenRouter.
|
A test prompt returns successfully through OpenRouter.
|
||||||
## Escalation
|
## Escalation
|
||||||
|
|||||||
+41
-17
@@ -3,8 +3,8 @@ title: Test OpenRouter
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
last_tested: 2026-10-20
|
last_tested: 2026-10-20
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [runbook, openrouter, providers, fallback]
|
tags: [runbook, openrouter, providers, fallback]
|
||||||
@@ -24,7 +24,8 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
|||||||
## Prerequisites
|
## Prerequisites
|
||||||
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
|
- Hermes CLI available: `hermes` on PATH (`/home/hermes/.local/bin/hermes` is a shim to the
|
||||||
uv-managed runtime — see the note under **Last tested**)
|
uv-managed runtime — see the note under **Last tested**)
|
||||||
- Internet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000
|
- Internet egress for opencode.ai, inference-api.nousresearch.com, kilo.ai (ollama.com is no
|
||||||
|
longer in the live chain — entry 3 uses the internal `guacano` endpoint instead)
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
@@ -32,22 +33,38 @@ Test whether OpenRouter is available to Hermes and whether fallback providers ar
|
|||||||
```bash
|
```bash
|
||||||
hermes fallback list
|
hermes fallback list
|
||||||
```
|
```
|
||||||
Expected active fallbacks on this install (live `hermes fallback list`, 2026-10-20 — **6** entries):
|
Expected active fallbacks on this install (live `hermes fallback list`, 2026-11-11 — **6** entries):
|
||||||
1. `mimo-v2.5` via opencode-go
|
1. `meituan/longcat-2.5-preview:free` via nous
|
||||||
2. `deepseek-v4-flash-free` via opencode-zen — `https://opencode.ai/zen/v1`
|
2. `deepseek-v4.1-flash` via opencode-zen — `https://opencode.ai/zen/v1`
|
||||||
3. `stepfun/step-3.7-flash:free` via nous — `https://inference-api.nousresearch.com/v1`
|
3. `deepseek-v4.1-flash` via custom:guacano — `http://ollama:11000/v1`
|
||||||
4. `qwen3.5:397b` via ollama-cloud — `https://ollama.com/v1`
|
4. `kilo-reflection-beam-free` via custom:litellm — `http://litellm:4000/v1`
|
||||||
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
|
5. `poolside/laguna-m.1:free` via kilocode — `https://api.kilo.ai/api/gateway`
|
||||||
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
|
6. `or-inclusionai_ling-2_6-1t` via custom — `http://litellm:4000/v1`
|
||||||
|
|
||||||
**Primary:** `meituan/longcat-2.5-preview:free` (via nous).
|
**Primary:** `smart-route` (via `custom:omni-101`, `http://100.78.63.20:20128`).
|
||||||
|
|
||||||
> **Changes since 2026-07-22:** `mimo-v2.5` via **opencode-go** has been **added at position 1** of the
|
> **Chain rewritten 2026-11-11 — the 2026-10-20 list recorded above was materially stale.**
|
||||||
> chain, and the primary changed to `meituan/longcat-2.5-preview:free` (was `freellmapi-auto`).
|
> Live `hermes fallback list` now reads (verbatim):
|
||||||
> The other five entries and their order are unchanged.
|
>
|
||||||
> `hermes fallback list` also prints a standing notice on every invocation:
|
> ```
|
||||||
> *"Claude Subscription DirectSDK (Experimental): Claude Code is not installed"* — expected on this
|
> Primary: smart-route (via custom:omni-101)
|
||||||
> box, since `claude` is not on PATH. It is a notice, not a failure of the fallback chain.
|
> Fallback chain (6 entries):
|
||||||
|
> 1. meituan/longcat-2.5-preview:free (via nous)
|
||||||
|
> 2. deepseek-v4.1-flash (via opencode-zen) [https://opencode.ai/zen/v1]
|
||||||
|
> 3. deepseek-v4.1-flash (via custom:guacano)
|
||||||
|
> 4. kilo-reflection-beam-free (via custom:litellm)
|
||||||
|
> 5. poolside/laguna-m.1:free (via kilocode) [https://api.kilo.ai/api/gateway]
|
||||||
|
> 6. or-inclusionai_ling-2_6-1t (via custom) [http://litellm:4000/v1]
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> What changed since the 2026-10-20 reading: the **primary moved back to OmniRoute**
|
||||||
|
> (`smart-route` via `custom:omni-101`, matching the live `model:` block — see
|
||||||
|
> [[concepts/model-routing]]); `mimo-v2.5` via opencode-go and `stepfun/step-3.7-flash:free` via
|
||||||
|
> nous have **left** the chain; `qwen3.5:397b` via ollama-cloud has **left** (so `ollama.com` no
|
||||||
|
> longer needs probing); `deepseek-v4.1-flash` via `custom:guacano` and `kilo-reflection-beam-free`
|
||||||
|
> via `custom:litellm` have **joined** at positions 3 and 4; positions 5 and 6 are unchanged.
|
||||||
|
> The standing *"Claude Subscription DirectSDK … Claude Code is not installed"* notice recorded on
|
||||||
|
> 2026-10-20 **did not print** on the 2026-11-11 run.
|
||||||
|
|
||||||
### 2. Test OpenRouter-style direct access if present
|
### 2. Test OpenRouter-style direct access if present
|
||||||
```bash
|
```bash
|
||||||
@@ -60,7 +77,6 @@ Current install state as of last run: OpenRouter is not configured as a standalo
|
|||||||
for url in \
|
for url in \
|
||||||
"https://opencode.ai/zen/v1/models" \
|
"https://opencode.ai/zen/v1/models" \
|
||||||
"https://inference-api.nousresearch.com/v1/models" \
|
"https://inference-api.nousresearch.com/v1/models" \
|
||||||
"https://ollama.com/v1/models" \
|
|
||||||
"https://api.kilo.ai/api/gateway/models"; do
|
"https://api.kilo.ai/api/gateway/models"; do
|
||||||
printf "%s -> " "$url"
|
printf "%s -> " "$url"
|
||||||
curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"
|
curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"
|
||||||
@@ -68,6 +84,13 @@ for url in \
|
|||||||
done
|
done
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Entries 3 (`custom:guacano` → `http://ollama:11000/v1`) and 6 (`custom` → `http://litellm:4000/v1`)
|
||||||
|
are **internal to the estate** and both now **require authentication**: a bare `curl` returns
|
||||||
|
**401 `invalid_api_key`**. Their keys live in `~/.hermes/.env` behind each provider's `key_env`
|
||||||
|
(e.g. `HERMES_CUSTOM_LITELLM_API_KEY`). `guacano` resolves through CT205 over the tailnet/DNS name
|
||||||
|
`ollama`, so it is only reachable from inside the estate — a 401 there proves auth reached the
|
||||||
|
node, not that it is down (see [[runbooks/provider-health]]).
|
||||||
|
|
||||||
### 4. Check Hermes fallback config reflects reality
|
### 4. Check Hermes fallback config reflects reality
|
||||||
```bash
|
```bash
|
||||||
grep -A5 "^fallback:" ~/.hermes/config.yaml
|
grep -A5 "^fallback:" ~/.hermes/config.yaml
|
||||||
@@ -89,7 +112,8 @@ Verify provider names and URLs in config match what the user wants to use.
|
|||||||
|
|
||||||
## Last tested
|
## Last tested
|
||||||
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
|
2026-07-22 — re-run 2026-10-20 against live `hermes fallback list` output (6 entries, primary
|
||||||
`meituan/longcat-2.5-preview:free`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
`meituan/longcat-2.5-preview:free`), and again on **2026-11-11** against the live output recorded
|
||||||
|
above (6 entries, primary `smart-route` via `custom:omni-101`). Note the CLI reports Hermes Agent **v0.22.2 (2026.9.24),
|
||||||
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
|
local `d923a5eb487a`**, and the `/home/hermes/.local/bin/hermes` launcher is a 73-byte shim that
|
||||||
delegates to the uv-managed runtime at
|
delegates to the uv-managed runtime at
|
||||||
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
|
`/home/hermes/.hermes/tools/python-3.14.7+20260901-linux-x64/` — *not* the
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Troubleshoot Tool Loop
|
|||||||
type: runbook
|
type: runbook
|
||||||
status: unresolved
|
status: unresolved
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
review_after: 2026-11-19
|
review_after: 2026-11-19
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [runbook, hermes]
|
tags: [runbook, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -94,6 +94,12 @@ delivered message contains real content.
|
|||||||
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
||||||
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
||||||
|
|
||||||
|
> **Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.**
|
||||||
|
> `tool_loop_guardrails`: `warnings_enabled: true`, `hard_stop_enabled: false`,
|
||||||
|
> `warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`,
|
||||||
|
> `hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`.
|
||||||
|
> `hard_stop_enabled: false` therefore still means a stuck tool loop only ever *warns*.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [[concepts/tool-calling]]
|
- [[concepts/tool-calling]]
|
||||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Cloudflare Tunnel
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, network, cloudflare]
|
tags: [system, network, cloudflare]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -33,6 +33,13 @@ Exposes internal homelab services, including Hermes, to the public internet with
|
|||||||
`hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119)
|
`hermes.martinwa.org` (9119), `hermes-chat.martinwa.org` (8787), `tg.hermes.martinwa.org` (19119)
|
||||||
and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]].
|
and `hermes-desktop.martinwa.org` (5174). Full live list in [[infrastructure/domains-and-tunnels]].
|
||||||
|
|
||||||
|
> **Ingress re-read live 2026-11-11 from `/etc/cloudflared/config.yml` on ct207 — unchanged.**
|
||||||
|
> `cloudflared.service` is still **active**, the tunnel is still `ct207-tunnel`, and the four
|
||||||
|
> Hermes-relevant entries are exactly as recorded: `hermes.martinwa.org`→`100.118.5.51:9119`,
|
||||||
|
> `hermes-chat.martinwa.org`→`100.118.5.51:8787`, `tg.hermes.martinwa.org`→`100.118.5.51:19119`,
|
||||||
|
> `hermes-desktop.martinwa.org`→`100.118.5.51:5174`. `mcp-portal.martinwa.org` is **still absent**
|
||||||
|
> from the ingress list — the 2026-10-20 "unresolved" flag stands, now three weeks old.
|
||||||
|
|
||||||
## Connected systems
|
## Connected systems
|
||||||
- [[systems/hermes-agent]] — the service actually reached through this tunnel
|
- [[systems/hermes-agent]] — the service actually reached through this tunnel
|
||||||
- [[infrastructure/domains-and-tunnels]] — full hostname table
|
- [[infrastructure/domains-and-tunnels]] — full hostname table
|
||||||
|
|||||||
+9
-2
@@ -3,8 +3,8 @@ title: Guanaco
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-14
|
created: 2026-08-14
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [llm, proxy, router, guanaco, ollama]
|
tags: [llm, proxy, router, guanaco, ollama]
|
||||||
sources: [live-probe, session-history]
|
sources: [live-probe, session-history]
|
||||||
@@ -40,6 +40,13 @@ LLM proxy/router providing model routing and tool-call formatting for the Ollama
|
|||||||
### v0.8.11 (previous)
|
### v0.8.11 (previous)
|
||||||
- Had known DSML tag leakage issues where malformed tool-call output could leak raw `<dsml_...>` syntax into model context stream
|
- Had known DSML tag leakage issues where malformed tool-call output could leak raw `<dsml_...>` syntax into model context stream
|
||||||
|
|
||||||
|
> **Live probe 2026-11-11 — the v0.8.15 record holds.** `/root/.guanaco/repo` on CT205
|
||||||
|
> (`pct exec 205`) still sits at tag **`v0.8.15`**, HEAD `5df435e3de11a51401972696d47daf9144b0d140`
|
||||||
|
> (the abbreviated `5df435e` recorded above), and the Ollama API on `localhost:11434/api/version`
|
||||||
|
> returns **200**. CT205 (`ollama`) is running; its LAN address is `192.168.178.158`.
|
||||||
|
> Note a naming nuance this page does not carry: Hermes's `guacano` provider points at
|
||||||
|
> **`http://ollama:11000/v1`**, i.e. Guanaco's own proxy port, not Ollama's `11434`.
|
||||||
|
|
||||||
## What DSML Is
|
## What DSML Is
|
||||||
DSML = "Daily Smart Messaging Log" — Guanaco's tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (`<dsml_invoke>`, bare `>` characters) into the conversation context, which confused downstream models.
|
DSML = "Daily Smart Messaging Log" — Guanaco's tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (`<dsml_invoke>`, bare `>` characters) into the conversation context, which confused downstream models.
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Headless Server
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [system, browser]
|
tags: [system, browser]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -39,6 +39,12 @@ still binds **all interfaces** — `0.0.0.0:5901` and `[::]:5901` — while both
|
|||||||
are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`).
|
are correctly loopback-only (`127.0.0.1:6080`, `127.0.0.1:6081`).
|
||||||
|
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 via `ps aux` + `ss -ltnp` — the live chain is exactly as recorded.**
|
||||||
|
> One `Xvfb :99` (`-screen 0 1600x1200x24 -ac -nolisten tcp`), one `x11vnc -display :99 …
|
||||||
|
> -rfbport 5901 -nopw` bound to `0.0.0.0:5901`/`[::]:5901`, one `fluxbox`, and two `websockify`
|
||||||
|
> instances (`127.0.0.1:6080`→`5900`, `127.0.0.1:6081`→`5901`). **Port 5900 still has no listener** —
|
||||||
|
> the 6080 pair still points at a dead port. Camofox's own listener is separately live on `*:9377`.
|
||||||
|
|
||||||
## Connected systems
|
## Connected systems
|
||||||
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
|
- [[systems/hermes-agent]] (browser backend uses this display — see [[systems/browser-backend]] / [[comparisons/camoufox-vs-chrome]])
|
||||||
- [[infrastructure/hosts]]
|
- [[infrastructure/hosts]]
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: hermes-ui (Browser Hermes Desktop)
|
|||||||
type: system
|
type: system
|
||||||
status: live
|
status: live
|
||||||
created: 2026-07-30
|
created: 2026-07-30
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, webui, desktop, browser]
|
tags: [hermes, webui, desktop, browser]
|
||||||
---
|
---
|
||||||
@@ -31,6 +31,12 @@ The official Hermes Desktop renderer, extracted from the `hermes-agent` monorepo
|
|||||||
> (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`,
|
> (the `/ui` Tailscale Serve route) — and **8787** is the Hermes WebUI (`python`, `0.0.0.0:8787`,
|
||||||
> the `/webui` route). Don't conflate 5174 / 5175 / 8787.
|
> the `/webui` route). Don't conflate 5174 / 5175 / 8787.
|
||||||
|
|
||||||
|
> **Both deployments re-verified 2026-11-11.** CT460: `0.0.0.0:5174` held by a `node` process
|
||||||
|
> (`ss -ltnp`), `/home/hermes/hermes-ui/` present. CT450: `ss -ltn` inside the container shows
|
||||||
|
> `0.0.0.0:5174` and `/home/hermes/hermes-ui` exists. The 5175/8787 distinction still holds:
|
||||||
|
> `127.0.0.1:5175` is a `python3` (the `/ui` Tailscale Serve route) and `0.0.0.0:8787` is the
|
||||||
|
> Hermes WebUI (`python`).
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
|
- Runs as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (`http://127.0.0.1:9119` by default).
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Nextcloud
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-08-17
|
created: 2026-08-17
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [nextcloud, file-sharing, self-hosted, infrastructure]
|
tags: [nextcloud, file-sharing, self-hosted, infrastructure]
|
||||||
sources: []
|
sources: []
|
||||||
@@ -40,6 +40,12 @@ Self-hosted file sync and sharing platform running on Proxmox LXC container.
|
|||||||
| news | 28.7.0 | active | RSS reader |
|
| news | 28.7.0 | active | RSS reader |
|
||||||
| nextcloud_announcements | 5.0.0 | active | |
|
| nextcloud_announcements | 5.0.0 | active | |
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — every fact on this page still holds.** CT270 (`nextcloud`) is running;
|
||||||
|
> `occ status` reports **installed: true, version 33.0.2.2, maintenance: false,
|
||||||
|
> needsDbUpgrade: false**; the container holds `192.168.178.56` (LAN) and `100.90.102.19`
|
||||||
|
> (Tailscale); `occ` lives at `/var/www/nextcloud/occ`. The `ncdownloader` NC33 incompatibility is
|
||||||
|
> unchanged and still the only app-level issue recorded.
|
||||||
|
|
||||||
## Known Issues
|
## Known Issues
|
||||||
- **ncdownloader v1.0.23 incompatible with Nextcloud 33 (2026-08-17):** App calls `OC\Server::getAppConfig()` which was removed in NC33. Crashes the `/settings/admin/overview` page with a fatal error. Fix: disable the app via `occ app:disable ncdownloader`. No data loss — ncdownloader settings preserved in DB, app just needs a compatible update.
|
- **ncdownloader v1.0.23 incompatible with Nextcloud 33 (2026-08-17):** App calls `OC\Server::getAppConfig()` which was removed in NC33. Crashes the `/settings/admin/overview` page with a fatal error. Fix: disable the app via `occ app:disable ncdownloader`. No data loss — ncdownloader settings preserved in DB, app just needs a compatible update.
|
||||||
|
|
||||||
|
|||||||
+7
-2
@@ -3,8 +3,8 @@ title: Obsidian Vault
|
|||||||
type: systems
|
type: systems
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-26
|
created: 2026-07-26
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [obsidian, notes, vault, homelab]
|
tags: [obsidian, notes, vault, homelab]
|
||||||
---
|
---
|
||||||
@@ -35,6 +35,11 @@ Personal knowledge base — notes, research, projects, and agent documentation.
|
|||||||
> (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`).
|
> (`obsidian-livesync`) on the `m93p` host** (`/opt/obsidian-livesync/`, holds only a `.env`).
|
||||||
> Full folder table, left for reference:
|
> Full folder table, left for reference:
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11 — the 2026-10-20 correction still holds.** `/home/hermes/obsidian-vault/`
|
||||||
|
> still contains only `50 - Research/` and `Cron Outputs/`, and is still **not** a git repo.
|
||||||
|
> `/home/hermes/wiki` is still a **local-only** git repo (`git remote -v` empty) tracking the
|
||||||
|
> `agent-estate-wiki/` mirror plus `reports/uber-eats/`. No numbered-folder vault exists on this box.
|
||||||
|
|
||||||
## Folder Structure
|
## Folder Structure
|
||||||
| Folder | Purpose |
|
| Folder | Purpose |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: OpenRouter
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: medium
|
confidence: medium
|
||||||
tags: [system, openrouter, provider]
|
tags: [system, openrouter, provider]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -29,6 +29,10 @@ OmniRoute’s catalog includes provider names and model IDs consistent with Open
|
|||||||
> model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still
|
> model catalog**, not as directly-routable Hermes providers. Routing to OpenRouter therefore still
|
||||||
> goes through OmniRoute, exactly as recorded here.
|
> goes through OmniRoute, exactly as recorded here.
|
||||||
|
|
||||||
|
> **Re-verified 2026-11-11.** `custom_providers` in `/home/hermes/.hermes/config.yaml` still holds
|
||||||
|
> exactly five entries — `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101` — with
|
||||||
|
> **no `openrouter` entry**. The 2026-10-20 claim holds unchanged 22 days on.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
|
- **User-provided:** Previously used; now largely replaced by LiteLLM and OmniRoute paths
|
||||||
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
|
- **Verified:** Decision page `keep-openrouter-as-fallback` exists, indicating retain intent
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-11-10
|
updated: 2026-11-11
|
||||||
verified_on: 2026-11-10
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [skills, hermes]
|
tags: [skills, hermes]
|
||||||
sources: []
|
sources: []
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ title: Terminal Backend
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-22
|
created: 2026-07-22
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [terminal, shell, backend, persistence]
|
tags: [terminal, shell, backend, persistence]
|
||||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||||
@@ -28,6 +28,13 @@ Live `terminal:` block re-read from `/home/hermes/.hermes/config.yaml` on 2026-1
|
|||||||
`container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
|
`container_persistent: true`, `docker_mount_cwd_to_workspace: false`, `docker_run_as_host_user: true`.
|
||||||
`backend: local` and `persistent_shell: true` are unchanged since 2026-07-22.
|
`backend: local` and `persistent_shell: true` are unchanged since 2026-07-22.
|
||||||
|
|
||||||
|
> **Full `terminal:` block re-read 2026-11-11 — every value unchanged.** `backend: local`,
|
||||||
|
> `persistent_shell: true`, `timeout: 180`, `home_mode: auto`, `container_cpu: 1`,
|
||||||
|
> `lifetime_seconds: 300`, `container_memory: 5120`, `container_disk: 51200`,
|
||||||
|
> `container_persistent: true`, `docker_mount_cwd_to_workspace: false`,
|
||||||
|
> `docker_run_as_host_user: true`. One nomenclature correction: the working-directory key is spelled
|
||||||
|
> **`cwd`** (`cwd: /home/hermes/workspace/`), not `workdir` — the value itself is unchanged.
|
||||||
|
|
||||||
## Persistence
|
## Persistence
|
||||||
- Shell state (venv, env vars, aliases) persists across terminal calls
|
- Shell state (venv, env vars, aliases) persists across terminal calls
|
||||||
- Working directory fixed to `/home/hermes/workspace`
|
- Working directory fixed to `/home/hermes/workspace`
|
||||||
|
|||||||
@@ -3,9 +3,9 @@ title: Hermes Tool Search
|
|||||||
type: system
|
type: system
|
||||||
status: active
|
status: active
|
||||||
created: 2026-07-25
|
created: 2026-07-25
|
||||||
updated: 2026-10-20
|
updated: 2026-11-11
|
||||||
review_after: 2026-11-20
|
review_after: 2026-11-20
|
||||||
verified_on: 2026-10-20
|
verified_on: 2026-11-11
|
||||||
version_applies_to: Hermes Agent on CT460
|
version_applies_to: Hermes Agent on CT460
|
||||||
confidence: high
|
confidence: high
|
||||||
tags: [hermes, automation, system]
|
tags: [hermes, automation, system]
|
||||||
@@ -49,6 +49,11 @@ tools:
|
|||||||
request instead of waiting for the tool schemas to cross an automatic token
|
request instead of waiting for the tool schemas to cross an automatic token
|
||||||
threshold.
|
threshold.
|
||||||
|
|
||||||
|
> **Config re-read 2026-11-11 — byte-for-byte unchanged.** `tools.tool_search` in
|
||||||
|
> `~/.hermes/config.yaml` still reads `enabled: "on"`, `threshold_pct: 20`,
|
||||||
|
> `search_default_limit: 8`, `max_search_limit: 50`, `listing_max_tokens: 8`. `tools:` carries no
|
||||||
|
> other key. Tool Search was exercised throughout this stocktake session.
|
||||||
|
|
||||||
## Verified result
|
## Verified result
|
||||||
|
|
||||||
- 80 external tools were deferred in the production gateway.
|
- 80 external tools were deferred in the production gateway.
|
||||||
|
|||||||
Reference in New Issue
Block a user