From d981e30128e79d463b68fafbe736620dddf4af78 Mon Sep 17 00:00:00 2001 From: hermes Date: Sun, 11 Oct 2026 03:11:07 +0800 Subject: [PATCH] stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections Drift found and corrected against live sources (not guessed): - OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live); CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted. Hermes custom:omni resolves to omniroute -> CT600, not CT601. - Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401), registered as custom provider omni-101 (5384 models, default smart-route, carries skills_hub/free-stack). Not in pct list; identity unconfirmed. - Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults). Both return HTTP 200. Page now distinguishes them. - Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html); undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row). - Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed. VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted. - Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900 no longer listening; "Serve API" claim flagged unverified. - Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----, hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers), CT333 (9 containers), CT460 via sudo. - Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d. Flagged, not investigated (no root cause): - mycompose-db-1 on CT200 is in Restarting (1) - crash-looping. 62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions, comparisons, most runbooks) - not probed against a live source, so reported not faked. Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed. Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt. --- agents/nanobot.md | 30 ++++++-- agents/qwenpaw.md | 18 +++-- current-state.md | 15 ++++ infrastructure/docker-services.md | 36 +++++++-- infrastructure/proxmox.md | 61 +++++++-------- infrastructure/tailscale.md | 53 ++++++++++++-- log.md | 13 ++++ raw/stocktake/freshness-20261011.txt | 106 +++++++++++++++++++++++++++ systems/ignis.md | 18 ++++- systems/omniroute.md | 28 +++++-- systems/skills-index.md | 10 +-- 11 files changed, 318 insertions(+), 70 deletions(-) create mode 100644 raw/stocktake/freshness-20261011.txt diff --git a/agents/nanobot.md b/agents/nanobot.md index b58d16f..473fec8 100644 --- a/agents/nanobot.md +++ b/agents/nanobot.md @@ -3,8 +3,8 @@ title: Nanobot type: agent status: active created: 2026-07-22 -updated: 2026-08-17 -verified_on: 2026-08-17 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [agent, nanobot, homelab] sources: [live-probe, runtime-config] @@ -20,14 +20,28 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat - **Workspace:** `/home/nano/.nanobot/workspace` ## Current state -- **Status: ACTIVE.** Confirmed running as of 2026-08-17. -- **Gateway:** port `8765` (HTTP API + chat completions) -- **WebSocket bridge:** port `8766` -- **Serve API:** port `8900` (host `0.0.0.0`, requires `api_key` in config) -- **WhatsApp Bridge:** `whatsapp-bridge.service` active on ct333 -- **Model preset:** `freellmapi-auto` (current session); baseline context window 200,000 tokens +- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening. +- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765` +- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766` +- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed. +- **Service unit names:** earlier notes listed `whatsapp-bridge.service` — `systemctl is-active nanobot gateway` and `is-active gateway` are NOT valid here (both returned `inactive`). Nanobot runs as a user service under the `nanobot` user (`systemd --user`, PID 166) plus the `nanobot-gateway` process; it is **not** a system-level `nanobot.service`. Check the gateway port/process, not the systemd unit. + +## Confirmed running on CT333 (2026-10-11) +| Process | Owner | Port(s) | Notes | +|---|---|---|---| +| `nanobot-gateway` (PID 2259048) | — | 8765, 8766 | The gateway | +| `mnemosyne` MCP (`mnemosyne mcp --transport std…`) | nano | — | `venvs/mnemosyne-mcp` venv | +| `chroma_query_api.py` | nano | — | started Sep 23 | +| `agentmail-daemon.py` | nano | — | started Sep 23 | +| `readlater` Next dev server | nano | 3210 | started Sep 23 | + +Other listeners on CT333 include 8787, 8880, 8899, 8999, 3000, 5050, 8080 and the Tailscale 443. + +## Docker on CT333 (verified 2026-10-11) +`steel-browser`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`. ## Cron jobs +Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded: - `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST - `morning-quota-report` — daily at 07:30 AWST - `daily-quota-730am` — daily at 07:30 AWST diff --git a/agents/qwenpaw.md b/agents/qwenpaw.md index 3d0762b..da17d56 100644 --- a/agents/qwenpaw.md +++ b/agents/qwenpaw.md @@ -3,11 +3,11 @@ title: QwenPaw type: system status: active created: 2026-08-15 -updated: 2026-08-15 -verified_on: 2026-08-15 -confidence: medium +updated: 2026-10-11 +verified_on: 2026-10-11 +confidence: high tags: [qwenpaw, ct465, agent, paseo] -sources: [skill-inspection, config-reference] +sources: [live-probe, config-reference] --- # QwenPaw @@ -24,10 +24,12 @@ Multi-agent gateway platform. Runs on CT465 with its own workspace, agent profil ## Current State -- **Status:** Active +- **Status:** Active — `paseo.service` active on CT465, verified 2026-10-11 - **Default chat model:** OmniRoute Smart Route (`auto/smart`) - **Config format:** JSON (`agent.json` per workspace) - **Daemon:** `paseo.service` +- **Also listening on CT465:** PVE exporter (`100.96.150.69:9221`), a Python service on `:9876`, and local SMTP `:25` +- **CT400/CT465 identity:** confirmed CT465 = `qwenpaw`, running ## Operating Layers @@ -58,9 +60,11 @@ qwenpaw daemon status When adding an OpenAI-compatible provider from Proxmox: - Verify target by VMID and hostname before writing config -- CT600/`omniroute` is the OmniRoute service at `100.88.81.19:20128` -- CT601/`omni` is a separate guest at `100.93.204.4` +- CT600/`omniroute` is the **live** OmniRoute service at `100.88.81.19:20128` — verified serving 2026-10-11 +- CT601/`omni` is a separate guest at `100.93.204.4` — **stopped**, do not use +- A third instance `omniroute-1` at `100.78.63.20:20128` is also live; Hermes registers it as provider `omni-101` - Query `/v1/models` using the exact API key stored for QwenPaw, not a management key +- See [[systems/omniroute]] for the full live node table ## Related diff --git a/current-state.md b/current-state.md index c3d200a..d1483ed 100644 --- a/current-state.md +++ b/current-state.md @@ -14,6 +14,21 @@ sources: [live-config, live-cron-list, live-hermes-version] # Current State Updated: 2026-10-11 +## Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST) + +- **OmniRoute CT600/CT601 roles are INVERTED.** CT600 `omniroute` (100.88.81.19) is **running** and serving `:20128` (combo API returns live combos); CT601 `omni` (100.93.204.4) is **stopped** with `onboot=0` (Tailscale peer offline 54d). Hermes resolves `custom:omni` → `omniroute:20128` → **CT600**. → [[systems/omniroute]] +- **A third OmniRoute node exists:** `omniroute-1` at `100.78.63.20:20128`, live (401 = auth-required), registered in Hermes config as provider `omni-101` (5384 models, default `smart-route`). It carries `skills_hub` (model `free-stack`) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in `pct list`; identity unconfirmed. → [[systems/omniroute]] +- **Two Ignis instances exist, on different hosts.** CT460 `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`, serving 200) and CT200 `ignis-ignis-1` (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`, serving 200). Both healthy; they are separate deployments. → [[systems/ignis]] +- **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. → [[infrastructure/tailscale]] +- **Proxmox inventory rebuilt** from `pct list` + `qm list`: **CT475 `agentzero` and VM110 `omarchy` were missing entirely**; 282 `ubuntu-server-xfce` corrected to stopped; dead rows removed (310, 500, 104, 106, 101). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed, not deleted. → [[infrastructure/proxmox]] +- **Nanobot verified on CT333:** `nanobot-gateway` (PID 2259048) alive on 8765/8766. It is **not** a systemd `nanobot.service`/`gateway.service` (both probes return `inactive` — wrong unit names); it runs as the `nanobot` user service plus the gateway process. Port **8900 is no longer listening** — previous "Serve API" claim flagged unverified. → [[agents/nanobot]] +- **Docker services page corrected:** the old "Docker runtime not available" line was a CT460 socket artifact (`/var/run/docker.sock` is `root:docker srw-rw----`; `hermes` user not in group `docker`; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → [[infrastructure/docker-services]] +- **Mattermost CT255 confirmed stopped**, consistent with its Tailscale peer being offline 21d. Root cause of the unreachable endpoint still undetermined. +- **⚠ `mycompose-db-1` on CT200 is in `Restarting (1)`** — crash-looping. Flagged; root cause NOT investigated this pass. +- Stocktake artifacts: skills-index regenerated (**346 skills / 70 categories**), `raw/stocktake/freshness-20261011.txt` = **62 stale / 24 fresh / 7 no-date of 93 pages**. +- **Quartz rebuild still inert** (no `quartz.config.ts`) — `public/` was not refreshed. Carried open item; unchanged from prior days. +- 62 pages remain >21d stale and were **not** corrected this pass: they were not probed against a live source, so they stay flagged rather than rewritten. + ## Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST) _The manual 2026-10-10 stocktake entry follows beneath this one._ diff --git a/infrastructure/docker-services.md b/infrastructure/docker-services.md index 3b1541e..55b10a8 100644 --- a/infrastructure/docker-services.md +++ b/infrastructure/docker-services.md @@ -3,11 +3,11 @@ title: Docker Services type: infrastructure status: mixed created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 -confidence: medium +updated: 2026-10-11 +verified_on: 2026-10-11 +confidence: high tags: [docker, homelab, infrastructure] -sources: [raw/configs/hermes-config-sanitized.txt] +sources: [raw/configs/hermes-config-sanitized.txt, live-probe] --- # Docker Services @@ -16,8 +16,32 @@ sources: [raw/configs/hermes-config-sanitized.txt] Containerized services supporting Hermes and the homelab. ## Status -- Docker runtime **not available** on this host (`docker ps` returned unavailable) -- Compose files confirmed present but execution state unverified +Docker runs on **three hosts**, and CT460's socket is not readable by the agent user. This page previously claimed "Docker runtime not available" — that is only true for CT460's socket from the `hermes` user. Real state, verified 2026-10-11: + +| Host | Access | Containers running | +|---|---|---| +| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **~50** — main Docker host | +| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | 9 (steel-browser, gotify, readlater, kokoro-tts, edge-tts, onecli-postgres, …) | +| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` (confirmed running; inspect requires root) | + +## CT200 live inventory (verified 2026-10-11, ~50 containers) +`guacamole-guacamole-1`, `telegram-bot-api`, `ytdlp-web-player`, `homarr`, `overtchat-searxng`, `searxng`, `khoj-search-1`, `prowlarr`, `overtchat-app`, `postgres-newsku`, `guacamole-postgres-1`, `jellyfin`, `libre-closet`, `apprise-api`, `tt-rss-web-nginx-1`, `tt-rss-updater-1`, `tt-rss-app-1`, `tt-rss-db-1`, `audiobookshelf`, `qbittorrent`, `open-webui`, `readlater`, `readlater-extraction-worker`, `worldmonitor-redis`, `redis-cache`, `nextstep-db`, `nextstep-scheduler-1`, `nextstep-app`, `portainer`, `flaresolverr`, `sshwifty`, `8mblocal`, `save-restricted-bot`, `kokoro-tts`, `edge-tts`, `filebrowser`, `filebrowser1`, `worldmonitor-redis-rest`, `worldmonitor`, `overtchat-redis`, `chromadb`, **`ignis-ignis-1`**, `nc_app_talk_bot_ai`, `excalidraw-frontend`, `nc_app_context_agent`, `portkey-gateway-web-1`, `guacamole-guacd-1`, `lobe-chat`, `mycompose-saas-1`. + +**Watch this one:** `mycompose-db-1` is in `Restarting (1)` — a crash-looping database. Flagged for follow-up; root cause not investigated on 2026-10-11. + +**Note the duplicate Ignis:** `ignis-ignis-1` on CT200 is a separate deployment from CT460's `ignis`. See [[systems/ignis]]. + +## CT201 runtipi — Tailscale routes (verified 2026-10-11) +| Host | Path | Target | +|---|---|---| +| `runtipi.kangaroo-eel.ts.net:8443` (tailnet) | `/` | `http://127.0.0.1:8085` | +| | `/mgr` | `http://127.0.0.1:8084` | +| `runtipi.kangaroo-eel.ts.net` (tailnet) | `/` | `http://127.0.0.1:8085` | +| | `/mgr` | `http://127.0.0.1:8084` | +| | `/steel` | `http://127.0.0.1:5173` | +| | `/steel-api` | `http://127.0.0.1:3000` | + +**Correction:** this page previously said "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" and listed CT201's `:8443` → `localhost:8082` for Homepage. Live `tailscale serve status` on CT201 shows the target is `127.0.0.1:8085` on both `:8443` and the default host, plus a `/steel` (5173) and `/steel-api` (3000) pair. The `/steel-api` route does not strip the path prefix — use the container IP directly for API calls. ## Confirmed Compose Files diff --git a/infrastructure/proxmox.md b/infrastructure/proxmox.md index fe9ba72..e56d357 100644 --- a/infrastructure/proxmox.md +++ b/infrastructure/proxmox.md @@ -3,7 +3,8 @@ title: Proxmox VE type: infrastructure status: active created: 2026-07-26 -updated: 2026-08-13 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [proxmox, virtualization, infrastructure, homelab] --- @@ -20,43 +21,45 @@ Hypervisor hosting all VMs and LXC containers for the homelab. - **Primary user:** `Anthony@pam` (full admin) - **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve` -## VMs (QEMU) -| VMID | Name | Status | Purpose | -|---|---|---|---| -| 101 | debianvm | destroyed | Debian VM — destroyed; services migrated to CT200 | -| 100 | homeassistant | running | Home Assistant | -| 282 | ubuntu-server-xfce | running | Ubuntu desktop — Sunshine, Kubuntu | -| 403 | open-claw | running | OpenClaw agent | -| 401 | hermes-old | stopped | Legacy Hermes VM (deprecated) | +## Live LXC Inventory (verified 2026-10-11 via `pct list`) -## LXC Containers | VMID | Name | Status | Purpose | |---|---|---|---| -| 200 | docker | running | Main Docker host — 42+ containers | +| 103 | cliproxyapi | running | Cliproxy API | +| 200 | docker | running | Main Docker host — 50+ containers (CT460's docker.sock is NOT accessible to the agent user; use CT200 for fleet inspection) | | 201 | runtipi | running | Runtipi — Homepage, ntfy, etc. | -| 205 | ollama | running | Ollama LLM inference + Guanaco proxy (v0.8.14, updated 2026-08-14) | -| 207 | webserver | running | Web server — Vercel AI, manifest router | +| 205 | ollama | running | Ollama LLM inference | +| 207 | webserver | running | Web server | +| 209 | cloudreve | **stopped** | Cloudreve (deprecated) | | 215 | n8n | running | N8n automation | | 221 | gitea | running | Gitea git server | | 241 | ntfy | running | Ntfy push notifications | -|| 245 | openweb | running | Open WebUI (Jul 30 updated, memory 4096 MB) | -| 255 | mattermost | running | Mattermost chat | +| 245 | openweb | running | Open WebUI | +| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer last seen 21d ago) | | 260 | freshrss | running | FreshRSS feed reader | -| 270 | nextcloud | running | NextCloud | -| 299 | sftpgo | running | SFTPGo file server — security review 2026-08-09: connects as root, unpinned fingerprint, `//` root path. **Incident 2026-09-02:** tailscaled inside CT299 died; PVE pre-start hook then failed because stale `lxc-attach` PIDs pinned the cgroup. Cleared stale PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`. Restored, tailscale direct 192.168.178.92:41641. CT299 swap is over-provisioned (43Gi swap fully used at 24Gi/31Gi RAM) — consider reducing swap or trimming workload. | -| 310 | monitoring | stopped | Monitoring stack (deprecated) | +| 270 | nextcloud | running | Nextcloud (apache2 active) | +| 299 | sftpgo | running | SFTPGo file server | | 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw | -| 450 | hermesagent | running | Hermes Agent LXC | -| 460 | hermes | running | Hermes CT460 | -| 465 | qwenpaw | running | QwenPaw LLM | -| 500 | litellm | running | LiteLLM proxy | -| 555 | elementsynapse | stopped | Element Synapse (deprecated) | -| 600 | omniroute | stopped | OmniRoute model router — retired 2026-08-13, migrated to CT601 | -| 601 | omni | running | OmniRoute model router — successor to CT600, since 2026-08-13 | -| 103 | cliproxyapi | running | Cliproxy API | -| 104 | rclone | running | Rclone sync | -| 106 | glance | stopped | Glance dashboard | -| 209 | cloudreve | stopped | Cloudreve (deprecated) | +| 450 | hermesagent | running | Hermes Agent CT450 (Chaos Lab) | +| 460 | hermes | running | Hermes CT460 (production) | +| 465 | qwenpaw | running | QwenPaw + Paseo daemon (active) | +| 475 | agentzero | running | AgentZero (`100.106.54.94`) — **not previously recorded** | +| 555 | elementsynapse | **stopped** | Element Synapse (deprecated) — also retired from Tailscale (offline 81d) | +| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) | +| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d | + +## Live QEMU Inventory (verified 2026-10-11 via `qm list`) + +| VMID | Name | Status | Notes | +|---|---|---|---| +| 100 | homeassistant | running | Home Assistant (13GB RAM) | +| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** | +| 282 | ubuntu-server-xfce | **stopped** | Ubuntu desktop | +| 310 | cachyos | **stopped** | CachyOS | +| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) | +| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed | + +**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry. ## API Tokens - `homepage@pve!homepage` — PVEAuditor role, privsep=0 (used by Homepage dashboard) diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 0e2dc26..0f0666d 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-10 -verified_on: 2026-10-10 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -16,9 +16,47 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access ## Domain - **Tailnet:** `kangaroo-eel.ts.net` -- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`) +- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) +## Tailscale Node Inventory (verified 2026-10-11) +Full node list from `tailscale status`. Offline peers are listed with their last-seen age. + +| Node | IP | State | +|---|---|---| +| hermes | 100.118.5.51 | **self** | +| agentzero | 100.106.54.94 | online | +| anthonys-imac | 100.112.123.124 | offline 17d | +| anthonys-ipad | 100.111.198.120 | online | +| anthonys-iphone | 100.100.116.67 | active | +| anthonys-mac-mini | 100.68.34.66 | offline 8d | +| cachyos | 100.124.208.15 | offline 37d | +| clawtest | 100.122.189.81 | online (CT333) | +| cliproxyapi | 100.83.112.94 | online (CT103) | +| cloudreve | 100.111.187.7 | offline 78d | +| conduit | 100.81.79.87 | offline 34d | +| docker | 100.96.150.69 | online (CT200) | +| docker-01 | 100.121.12.31 | online | +| elementsynapse | 100.75.250.37 | offline 81d | +| freellmapi | 100.127.208.75 | online | +| freshrss | 100.83.60.93 | online | +| freshrss-new | 100.110.180.115 | offline 54d | +| gitea | 100.82.100.128 | online (CT221) | +| hermesagent | 100.98.205.112 | online (CT450) | +| homeassistant | 100.68.139.5 | active, exit node | +| litellm | 100.79.29.64 | online | +| m93p | 100.104.51.98 | online | +| mailflow | 100.115.60.92 | online | +| mattermost | 100.98.116.14 | **idle/offline 21d** — CT255 is stopped, matches [[infrastructure/proxmox]] | +| n8n | 100.84.124.30 | online | +| nextcloud | 100.90.102.19 | online (CT270) | +| ntfy | 100.67.153.116 | active | +| ollama | 100.108.44.119 | online (CT205) | +| omarchy-macmini | 100.104.53.105 | offline 8d | +| omarchy-quattro | 100.106.27.27 | offline 40d | + +> Note: Tailscale's offline list is not authoritative for container state — cross-check `pct list` on the PVE host. + ## CT460 Serve Routes | Path | Target | Notes | |---|---|---| @@ -32,13 +70,16 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | | `/synergy-prep` | `http://127.0.0.1:9123/` | Synergy prep | -| `/agent-estate` | `http://127.0.0.1:8769/agent-estate-role-dashboard.html` | Role dashboard | +| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | -## CT460 Funnel Routes +Verified against `tailscale serve status` on 2026-10-11. + +## CT460 Funnel and 9443 Routes | Port | Target | Notes | |---|---|---| -| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel | +| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) | +| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 | ## CT201 Serve Routes | Path | Target | Notes | diff --git a/log.md b/log.md index 0c71b67..0124c5d 100644 --- a/log.md +++ b/log.md @@ -459,3 +459,16 @@ neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined. `incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. + +## [2026-10-11] stocktake | Daily maintenance + live drift corrections +- Ran `wiki-maintain.sh`: skills-index regenerated (346 skills / 70 categories), freshness audit written to `raw/stocktake/freshness-20261011.txt` (62 stale / 24 fresh / 7 no-date of 93 pages) +- **OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed.** Verified via `pct list`, `pct config`, `ss` and `tailscale status`: CT600 `omniroute` (100.88.81.19) is RUNNING and serving `:20128`; CT601 `omni` (100.93.204.4) is STOPPED with `onboot=0`. Hermes resolves `custom:omni` → `omniroute:20128` → CT600, not CT601. systems/omniroute.md rewritten with the live node table. +- **Third OmniRoute node found:** `omniroute-1` at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider `omni-101` (5384 models, default `smart-route`, carries `skills_hub`). Not in `pct list` — identity unconfirmed. +- **Ignis duplicate found:** a second `ignis-ignis-1` container runs on CT200 (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`), separate from CT460's `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them. +- **Tailscale route table corrected** against live `tailscale serve status`: `/agent-estate` target is bare `http://127.0.0.1:8769` (not `.../8769/agent-estate-role-dashboard.html`); previously-missing tailnet-only `:9443` → `127.0.0.1:5173` documented. CT201 routes refreshed (8085 targets + `/steel` 5173 + `/steel-api` 3000). +- **Proxmox inventory rebuilt** from live `pct list` + `qm list`: added CT475 `agentzero` and VM110 `omarchy`; corrected 282 `ubuntu-server-xfce` to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed rather than guessed. +- **Nanobot verified** on CT333: gateway `nanobot-gateway` (PID 2259048) listening on 8765/8766; NOT a systemd `nanobot.service`/`gateway.service` (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified. +- **Docker services page corrected:** the old "Docker runtime not available" note was a CT460 socket-permission artifact (`/var/run/docker.sock` is `root:docker srw-rw----`, `hermes` user not in group `docker`, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo). +- **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping. +- Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d. +- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten. diff --git a/raw/stocktake/freshness-20261011.txt b/raw/stocktake/freshness-20261011.txt new file mode 100644 index 0000000..8951275 --- /dev/null +++ b/raw/stocktake/freshness-20261011.txt @@ -0,0 +1,106 @@ +# Wiki freshness audit — 2026-10-11 +Threshold: >21 days since `updated` + +## STALE (62) + +- [ 81d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22) +- [ 81d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22) +- [ 81d] concepts/agent-memory.md (updated 2026-07-22) +- [ 81d] concepts/model-routing.md (updated 2026-07-22) +- [ 81d] concepts/persistent-shells.md (updated 2026-07-22) +- [ 81d] concepts/tool-calling.md (updated 2026-07-22) +- [ 81d] decisions/index.md (updated 2026-07-22) +- [ 81d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22) +- [ 81d] decisions/persistent-shell-enabled.md (updated 2026-07-22) +- [ 81d] experiments/index.md (updated 2026-07-22) +- [ 81d] infrastructure/backups.md (updated 2026-07-22) +- [ 81d] infrastructure/credentials-map.md (updated 2026-07-22) +- [ 81d] infrastructure/docker-services.md (updated 2026-07-22) +- [ 81d] infrastructure/domains-and-tunnels.md (updated 2026-07-22) +- [ 81d] infrastructure/network-map.md (updated 2026-07-22) +- [ 81d] queries/retained-research-answers.md (updated 2026-07-22) +- [ 81d] runbooks/backup-wiki.md (updated 2026-07-22) +- [ 81d] runbooks/diagnose-docker.md (updated 2026-07-22) +- [ 81d] runbooks/recover-docker-service.md (updated 2026-07-22) +- [ 81d] runbooks/restart-browser.md (updated 2026-07-22) +- [ 81d] runbooks/restart-hermes.md (updated 2026-07-22) +- [ 81d] runbooks/restore-openrouter.md (updated 2026-07-22) +- [ 81d] runbooks/test-openrouter.md (updated 2026-07-22) +- [ 81d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22) +- [ 81d] systems/cloudflare-tunnel.md (updated 2026-07-22) +- [ 81d] systems/headless-server.md (updated 2026-07-22) +- [ 81d] systems/home-assistant.md (updated 2026-07-22) +- [ 81d] systems/openrouter.md (updated 2026-07-22) +- [ 81d] systems/terminal-backend.md (updated 2026-07-22) +- [ 78d] runbooks/gateway-resource-resilience.md (updated 2026-07-25) +- [ 78d] systems/tool-search.md (updated 2026-07-25) +- [ 77d] systems/homepage-dashboard.md (updated 2026-07-26) +- [ 77d] systems/obsidian.md (updated 2026-07-26) +- [ 75d] runbooks/job-radar.md (updated 2026-07-28) +- [ 75d] runbooks/multiplexer-setup.md (updated 2026-07-28) +- [ 74d] systems/opencode-go.md (updated 2026-07-29) +- [ 73d] systems/hermes-ui.md (updated 2026-07-30) +- [ 71d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01) +- [ 70d] comparisons/steel-vs-camoufox.md (updated 2026-08-02) +- [ 70d] concepts/browser-automation.md (updated 2026-08-02) +- [ 70d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02) +- [ 70d] reports/uber-eats/index.md (updated 2026-08-02) +- [ 70d] systems/browser-backend.md (updated 2026-08-02) +- [ 65d] systems/nanobot.md (updated 2026-08-07) +- [ 60d] comparisons/memory-backends.md (updated 2026-08-12) +- [ 60d] comparisons/model-routers.md (updated 2026-08-12) +- [ 59d] infrastructure/proxmox.md (updated 2026-08-13) +- [ 58d] systems/guanaco.md (updated 2026-08-14) +- [ 58d] systems/omniroute.md (updated 2026-08-14) +- [ 57d] agents/index.md (updated 2026-08-15) +- [ 57d] agents/open-claw.md (updated 2026-08-15) +- [ 57d] agents/qwenpaw.md (updated 2026-08-15) +- [ 57d] agents/zeroclaw.md (updated 2026-08-15) +- [ 57d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15) +- [ 56d] CONTRIBUTING.md (updated 2026-08-16) +- [ 55d] agents/nanobot.md (updated 2026-08-17) +- [ 55d] systems/nextcloud.md (updated 2026-08-17) +- [ 54d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-08-18) +- [ 53d] systems/ignis.md (updated 2026-08-19) +- [ 44d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28) +- [ 43d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29) +- [ 37d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04) + +## FRESH (24) + +- [ 7d] agents/hermes-chaos.md +- [ 7d] agents/hermes-production.md +- [ 0d] current-state.md +- [ 8d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 0d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 5d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 7d] incidents/2026-10-03-ct450-hermes-update.md +- [ 7d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 5d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 2d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 2d] incidents/2026-10-09-memory-recall-relevance.md +- [ 7d] infrastructure/hosts.md +- [ 1d] infrastructure/tailscale.md +- [ 7d] runbooks/provider-health.md +- [ 7d] runbooks/update-hermes-safely.md +- [ 9d] systems/current-profile.md +- [ 0d] systems/hermes-agent.md +- [ 7d] systems/memory-backend.md +- [ 0d] systems/messaging-integrations.md +- [ 0d] systems/mnemosyne.md +- [ 10d] systems/model-providers.md +- [ 0d] systems/paseo.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md + +## NO updated DATE (7) + +- SCHEMA.md +- experiments/active/example.md +- incidents/2026-07-29-mnemosyne-relay.md +- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- index.md +- log.md +- reports/uber-eats/2026-07.md + +TOTAL: 62 stale, 24 fresh, 7 no-date of 93 pages \ No newline at end of file diff --git a/systems/ignis.md b/systems/ignis.md index ac06ba0..9414a13 100644 --- a/systems/ignis.md +++ b/systems/ignis.md @@ -3,8 +3,8 @@ title: Ignis type: system status: active created: 2026-08-19 -updated: 2026-08-19 -verified_on: 2026-08-19 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [ignis, obsidian, browser-native, wiki, docker] sources: [live-probe, docker-inspect] @@ -18,10 +18,12 @@ Browser-native Obsidian interface for the agent estate wiki. Provides full Obsid ## Details - **Image:** `ignis-subpath:0.8.9` (custom build with subpath support) - **Container:** `ignis` on CT460 (Hermes LXC) -- **Port:** `8080` (bound to `0.0.0.0`) +- **Port:** `8080` (bound to `0.0.0.0`) — confirmed `HTTP 200` on `http://127.0.0.1:8080/` 2026-10-11 - **Tailscale Serve:** `https://hermes.kangaroo-eel.ts.net/ignis` → `http://127.0.0.1:8080` - **Privileged:** No (subpath build handles glibc threading without it) - **Restart:** `unless-stopped` +- **Server process:** `node /app/apps/ignis-server/server/index.js` running as user `hermes` (PID observed 875) under `containerd-shim-runc-v2`, container `453483b8d99c…` +- **Docker socket:** `/var/run/docker.sock` is `root:docker` `srw-rw----` — the `hermes` user is not in group `docker`, so `docker ps`/`docker inspect` on CT460 require `sudo` (no passwordless sudo in this cron context). Verify Ignis via the HTTP endpoint and process list instead. ## Volume Mounts | Host Path | Container Path | Purpose | @@ -38,6 +40,16 @@ Browser-native Obsidian interface for the agent estate wiki. Provides full Obsid ## Docker Compose Located at `/home/hermes/docker/ignis/docker-compose.yml` on CT460. +## ⚠ Second Ignis instance on CT200 (found 2026-10-11) +There is a **separate** `ignis-ignis-1` container running on **CT200** (the Docker host), not CT460: +- Image `ignis-server:latest`, state running +- Host port **8092** → container 8080 +- Vault mount `/opt/ignis/vaults` → `/vaults` (different vault path from CT460's `/home/hermes/wiki/agent-estate-wiki`) +- Data mount `/opt/ignis/data` → `/app/data` +- Obsidian version 1.12.7, Node 22.23.1 + +**Do not conflate the two.** The canonical wiki-serving Ignis for the agent estate wiki is the CT460 instance exposed at `/ignis`. The CT200 instance is a different deployment on a different vault. Confirm which instance a user means before troubleshooting. + ## Access - **Tailscale (primary):** `https://hermes.kangaroo-eel.ts.net/ignis` - **Local (on CT460):** `http://localhost:8080` diff --git a/systems/omniroute.md b/systems/omniroute.md index f3d7903..857327c 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-08-14 -verified_on: 2026-08-14 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -16,16 +16,32 @@ sources: [raw/configs/hermes-config-sanitized.txt] Secondary provider/router for Hermes auxiliary roles and some agent routing. ## Current Configuration -- **Hermes provider name:** `custom:omni` -- **Base URL:** `http://omni:20128/v1` -- **Host reachability:** `omni.kangaroo-eel.ts.net` resolves to `100.93.204.4`; CT601 running, direct HTTP reachable -- **Runs on:** Proxmox CT601 (`omni`) — migrated from CT600 (`omniroute`, now stopped) on 2026-08-13 +- **Hermes provider name:** `custom:omni` (maps to provider `omniroute:20128`) +- **Base URL:** `http://omniroute:20128/v1` +- **Host reachability:** bare name `omniroute` resolves to `100.88.81.19`; direct HTTP on `:20128` answers `401` (auth-required, service live) +- **Runs on:** Proxmox CT600 (`omniroute`, Tailscale `100.88.81.19`, onboot=1) — verified running 2026-10-11 - **DB path:** `/opt/omniroute/storage.sqlite` (was `/var/lib/omniroute/storage.sqlite`) - **Backup:** pre-migration config saved at `~/.hermes/config.yaml.bak.omniroute-removal` - **Miss:** Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead - **Auxiliary roles in Hermes config:** `skills_hub`, `approval`, `mcp` - **Key prefix visible in config:** `sk-5c67fbdc0...` — full value not stored in wiki +## CT600 / CT601 / omniroute-1 — actual state (2026-10-11) + +The wiki previously recorded CT600 as retired and CT601 as the live successor. **That is no longer true.** Live PVE and Tailscale checks return: + +| Node | Status (live 2026-10-11) | Tailscale IP | Notes | +|---|---|---|---| +| CT600 `omniroute` | **running**, onboot=1 | `100.88.81.19` | OmniRoute listening on `0.0.0.0:20128`; `/v1/models` returns `401` (alive, auth required) | +| CT601 `omni` | **stopped**, onboot=0 | `100.93.204.4` | Tailscale peer last seen 54d ago | +| `omniroute-1` | **active** (not in PVE `pct list`) | `100.78.63.20` | Third OmniRoute node; `/v1/models` returns `401`. Registered in Hermes config as custom provider `omni-101`, which carries `skills_hub` (model `free-stack`, 5384 models, default `smart-route`) | + +- Hermes config resolving to the omniroute hostname (`custom:omni` → `omniroute:20128`) therefore points at **CT600**, not CT601. +- CT600's combo API is reachable with its local key and returns combos (e.g. `go-pro`), confirming it is the serving instance. +- **Do not treat CT601 as the production OmniRoute node.** It is stopped with `onboot=0`. +- Treat `omniroute-1` (`100.78.63.20`) as a second live instance used by the `omni-101` provider. It is not visible in `pct list`, so it is likely a VM or off-PVE host; its identity was not confirmed. + + ## Catalog and Routing Facts - Catalog reachable: yes, 47 combos returned at `/api/combos` - Combo schema observed: `name`, `strategy`, `models`, `id`; some combos have `config`, `isHidden`, `uuid` diff --git a/systems/skills-index.md b/systems/skills-index.md index fa0adba..abd7f88 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-10 -verified_on: 2026-10-10 +updated: 2026-10-11 +verified_on: 2026-10-11 confidence: high tags: [skills, hermes] sources: [] @@ -20,7 +20,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - Category = directory structure under `~/.hermes/skills/` (`category/skill/SKILL.md`). - Skills directly under `~/.hermes/skills/` group under `uncategorized`. -## .Archive (39) +## .Archive (40) - `agentmail-webhook-tailscale` — # AgentMail Real-Time Webhook via Tailscale Funnel - `anthropic-spend-forensics` — # Anthropic Spend Forensics @@ -40,6 +40,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `hermes-multiplex-telegram-ops` — # Hermes Multiplex Telegram Ops - `hermes-profile-scoped-platform-troubleshooting` — # Hermes Profile-Scoped Platform Troubleshooting - `hermes-session-context-ops` — # Hermes Session-Context Operations +- `hermes-ui-ops` — # Hermes UI Operations - `hermes-web-backend-config` — # Hermes Web Backend Config - `homelab-migration-planning` — # Test - `in-repo-documentation` — # In-Repository Documentation @@ -326,12 +327,11 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit. - `groktocrawl-deployment` — # GroktoCrawl Self-Hosted Deployment -## Hermes (4) +## Hermes (3) - `hermes-personality-soulmd` — # Hermes Personality & SOUL.md - `hermes-telegram-miniapp` — # Hermes Telegram Mini App — Deploy & Operate - `hermes-tts-configuration` — # Hermes TTS & Voice Configuration -- `hermes-ui-ops` — # Hermes UI Operations ## Hermes Api Vault (1)