wiki: nightly audit 2026-11-05 window — network-map re-verified by live probe (9377=Camofox not Stratos, 8790/5174 gone, +9120/9222/7777/6767/8085/9900, full Tailscale Serve table, two dead routes 8769/9123); commit pending 11-04 stocktake re-verifications; log + skills-index
This commit is contained in:
@@ -3,8 +3,8 @@ title: Backups
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-22
|
||||
verified_on: 2026-10-22
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: medium
|
||||
tags: [infrastructure, backups]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
@@ -59,7 +59,7 @@ How Hermes's own configuration/state gets backed up, and what's missing.
|
||||
`e27da6c34bbb` (**Hermes Config Backup to Gitea**, daily `0 11 * * *`, `hermes-config-backup.sh`) **commits
|
||||
locally every night and fails to push every night — while its cron row still reads `last_status: ok` and its
|
||||
execution row reads `completed`.** Last commit that reached Gitea: **2026-08-07 11:00:47**; the local repo is
|
||||
**76 commits ahead of `origin/main`**. Root cause is a credential: `origin` is
|
||||
**89 commits ahead of `origin/main`** (was 76 on 2026-10-22 — **13 more nights of local-only commits** since). Root cause is a credential: `origin` is
|
||||
`https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` — HTTPS, no userinfo, no credential
|
||||
helper — so the non-interactive push dies on
|
||||
`fatal: could not read Username … No such device or address` → `✗ Push to Gitea FAILED` →
|
||||
@@ -71,6 +71,27 @@ since 2026-08-07.** The 2026-07-22 full zip and the wiki/authoring repos remain
|
||||
Telegram failure notices *have* been firing daily (last: 2026-10-21 11:00:14, `message_id 8449`); the AgentMail
|
||||
notification step failed.
|
||||
|
||||
**Re-verified 2026-11-04 — still dead, and the drift is now 89 commits.** Live `git -C ~/.hermes/config-backup`:
|
||||
`origin` = `https://gitea.kangaroo-eel.ts.net/Anthony/hermes-config-backup.git` (**unchanged** — still HTTPS with
|
||||
no userinfo and no credential helper, so the credential root cause has not been addressed); `rev-list --count
|
||||
origin/main..HEAD` = **89**; `log origin/main -1` = **2026-08-07 11:00:47 +0800 1e1eb208** (`🦝 nightly backup:
|
||||
2026-08-07 11:00:29 AWST`) — i.e. **nothing has reached Gitea in 89 days**. `backup.log` confirms the nightly
|
||||
failure is current, not historical: the last two runs read
|
||||
`[2026-11-02 11:00:32 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---` and
|
||||
`[2026-11-03 11:00:31 AWST] ✗ Push to Gitea FAILED` / `--- Backup complete: FAIL ---`.
|
||||
**A second, separate push failure is now also present:** the script attempts a `github` remote that **does not
|
||||
exist** — `fatal: 'github' does not appear to be a git repository` → `✗ Push to GitHub FAILED`. So both off-box
|
||||
targets are broken; the GitHub leg is a missing-remote error rather than a credential error, and would need
|
||||
`git remote add github …` with credentials before it can work at all.
|
||||
**The cron row still lies:** `e27da6c34bbb` reads `enabled: true, last_status: ok` in `jobs.json` while every
|
||||
nightly execution has failed the push — the classic green-row/dead-artifact pattern.
|
||||
|
||||
**Full-backup gap unchanged:** the one full `hermes backup` zip on disk is still
|
||||
`/home/hermes/hermes-backup-2026-07-22-180830.zip` (246 MB, 22 Jul) — **now 105 days old**. No new full zip has
|
||||
been taken. `crontab -l -u hermes` is still empty and there are still **0 user systemd timers**, so nothing
|
||||
schedules the full zip; only `e5ed215fc91c` (**Mnemosyne DB Backup**, weekly `0 3 * * 0`) and `e27da6c34bbb`
|
||||
(config backup, daily) exist in the cron fleet, and neither takes a full `hermes backup`.
|
||||
|
||||
## Relevant runbooks
|
||||
- [[runbooks/backup-wiki]]
|
||||
- [[runbooks/update-hermes-safely]]
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Docker Services
|
||||
type: infrastructure
|
||||
status: mixed
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
confidence: high
|
||||
tags: [docker, homelab, infrastructure]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe]
|
||||
@@ -20,14 +20,41 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u
|
||||
|
||||
| Host | Access | Containers running |
|
||||
|---|---|---|
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** — main Docker host (**verified 2026-10-31**) |
|
||||
| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **62** — main Docker host (**re-verified 2026-11-04**) |
|
||||
| CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**verified 2026-10-31**) |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** — `thirsty_payne`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29** |
|
||||
| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** — `nostalgic_leavitt`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` was **replaced 2026-11-02** by `nostalgic_leavitt` (`ghcr.io/github/github-mcp-server`, created 2026-11-02) |
|
||||
| CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented |
|
||||
| CT460 `hermes` | `sudo docker` — socket is `root:docker srw-rw----`, `hermes` user not in group `docker`; **no passwordless sudo in cron** | `ignis` — **running** (`Up 35 hours`), verified via `pct exec 460 -- docker ps` from the PVE host |
|
||||
| CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 4 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-04 |
|
||||
|
||||
## CT200 live inventory (re-verified 2026-11-01, 61 containers)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
## CT200 live inventory (re-verified 2026-11-04, 62 containers)
|
||||
`8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, **`worldmonitor-seeder-scheduled`**, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`.
|
||||
|
||||
**Re-verified 2026-11-04 — 61 → 62 containers, one addition.** A set-wise diff against live
|
||||
`docker ps --format '{{.Names}}'` on CT200 shows **exactly one container live but absent from the previous list:
|
||||
`worldmonitor-seeder-scheduled`** (image `worldmonitor:latest`, `Up 29 minutes`, **status `unhealthy`**). Every
|
||||
other name in the list above is confirmed live, and **no listed container has disappeared**. Note the seeder's
|
||||
**new** name — the 2026-11-01 correction removed a `worldmonitor-seeder-fast` entry as "not running", and live
|
||||
`docker ps -a --filter name=worldmonitor` now shows four worldmonitor containers: `worldmonitor` (`Up 4 days`,
|
||||
healthy), **`worldmonitor-seeder-scheduled`** (`Up 29 minutes`, **unhealthy**), `worldmonitor-redis` and
|
||||
`worldmonitor-redis-rest` (`Up 4 days`). The `worldmonitor-seeder-fast` container **does not exist at all** —
|
||||
not even stopped (`docker ps -a` lists only the four above), so the old entry was correctly removed and the seeder
|
||||
has since been redeployed under a new name. **Watch item:** `worldmonitor-seeder-scheduled` is reporting
|
||||
**unhealthy** — worth a look if it does not clear.
|
||||
|
||||
**CT333 container set — `thirsty_payne` → `nostalgic_leavitt` (2026-11-04).** Live CT333 `docker ps` returns 8
|
||||
running containers, and the name that had replaced `adoring_jemison` on 2026-11-01 has itself been replaced:
|
||||
`nostalgic_leavitt` (image `ghcr.io/github/github-mcp-server`, **created 2026-11-02T17:01:17Z**, `Up 29 hours`).
|
||||
The count is unchanged at 8 and the other seven names are identical. A ninth container, `qdrant`
|
||||
(`qdrant/qdrant`), exists but is **stopped** (`Exited (143) 4 months ago`) — it is not in the running count and
|
||||
is not listed above. **Docker generates these random names for compose-less containers, so repeated renames are
|
||||
normal — the reliable check is always the live set, never the recorded name.**
|
||||
|
||||
**CT460 `ignis` access path corrected (2026-11-04).** The old row said "no passwordless sudo in cron", which
|
||||
implied the container list could not be read at all. It can: **`pct exec 460 -- docker ps` from the PVE host
|
||||
runs as root and works** (`ignis|ignis-subpath:0.8.9|Up 4 days|0.0.0.0:8080->8080/tcp`). What does *not* work is
|
||||
the `hermes` user's own `docker ps` inside CT460 — `permission denied while trying to connect to the docker API
|
||||
at unix:///var/run/docker.sock` — because the socket is `root:docker srw-rw----` and `hermes` is not in group
|
||||
`docker`. Use the root-via-PVE path.
|
||||
|
||||
**Re-verified 2026-11-01:** the 2026-10-11 inventory listed **49** containers; the live count is now **61**.
|
||||
**Corrected 2026-11-01:** the inventory list had three **misspelled** `overtchat-*` entries (written
|
||||
|
||||
@@ -3,11 +3,11 @@ title: Network Map
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-14
|
||||
verified_on: 2026-10-14
|
||||
updated: 2026-11-05
|
||||
verified_on: 2026-11-05
|
||||
confidence: high
|
||||
tags: [infrastructure, network]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05]
|
||||
---
|
||||
|
||||
# Network Map
|
||||
@@ -22,11 +22,13 @@ Where Hermes and its dependencies run, and how traffic reaches them.
|
||||
- **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0)
|
||||
|
||||
### Tailscale (`kangaroo-eel.ts.net`)
|
||||
_Peer IPs below re-checked 2026-11-05 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed._
|
||||
- **pve:** `100.96.100.82`
|
||||
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
|
||||
- **hermes (ct460):** `100.118.5.51`
|
||||
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
|
||||
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
|
||||
- **omniroute-1 (third OmniRoute node):** `100.78.63.20` (`omniroute-1.kangaroo-eel.ts.net`) — added 2026-11-05.
|
||||
- **nanobot (ct333, "clawtest"):** `100.122.189.81`
|
||||
- **webserver (ct207, runs the Cloudflare Tunnel):** `100.82.40.23`
|
||||
- **debianvm (VM 101):** `100.98.95.65`
|
||||
@@ -42,19 +44,51 @@ Where Hermes and its dependencies run, and how traffic reaches them.
|
||||
| Port | Bind | Process | Purpose |
|
||||
|---|---|---|---|
|
||||
| 22 | `*` | sshd | SSH |
|
||||
| 80 / [::]:80 | `*` | nginx | reverse proxy, default vhost |
|
||||
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (tunneled) |
|
||||
| 9119 | `0.0.0.0` | hermes (webui) | Hermes Web UI backend |
|
||||
| 8644 / 8642 | `*` | hermes | additional Hermes agent ports |
|
||||
| 5174 | `*` | node | Hermes desktop companion (tunneled as hermes-desktop.martinwa.org) |
|
||||
| 9377 | `*` | node | Stratos — AI Command Centre |
|
||||
| 8787 | `127.0.0.1`* | python | Hermes gateway health endpoint (see [[runbooks/update-hermes-safely]]) |
|
||||
| 9099 / 8790 | `*` | python3 | auxiliary Hermes processes (unverified purpose) |
|
||||
| 5901 / [::]:5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
|
||||
| 6080 / 6081 | `127.0.0.1` | websockify | noVNC web bridges for two VNC displays |
|
||||
| 80 | `*` | nginx | reverse proxy, default vhost |
|
||||
| 443 / 8443 / 9443 | `100.118.5.51` (tailscale) | tailscaled | Tailscale Serve/Funnel listeners — route table below |
|
||||
| 19119 | `*` | nginx → 127.0.0.1:9119 | public Hermes webui path (Cloudflare-tunneled) |
|
||||
| 9119 | `0.0.0.0` | `hermes-webui/server.py` | Hermes Web UI backend (Serve `/`) |
|
||||
| 8644 / 8642 | `*` | hermes (gateway) | additional Hermes agent ports |
|
||||
| 9900 | `0.0.0.0` | hermes (gateway) | gateway aux port (up 2026-11-05) |
|
||||
| 5174 | — | *(no listener)* | desktop companion port listed up to 2026-10-14; **not listening 2026-11-05** |
|
||||
| 5173 | `0.0.0.0` | node (vite) | content-spark-hub dev server (Serve `:9443` `/`) |
|
||||
| 5175 | `127.0.0.1` | `python3 -m http.server` (`hermes-ui/app/dist`) | hermes-ui static build (Serve `/ui`) |
|
||||
| 9377 | `*` | node — `/home/hermes/services/camofox-browser` | **Camofox browser REST** (`{"ok":true,"engine":"camoufox"}`). Corrected 2026-11-05 — this port had been recorded as "Stratos — AI Command Centre". |
|
||||
| 9222 | `0.0.0.0` | node `camoufox-ws-server.mjs` | Camofox Playwright/CDP WebSocket endpoint |
|
||||
| 7777 | `0.0.0.0` | node `@openchamber/web serve` | OpenChamber web |
|
||||
| 6767 | `0.0.0.0` | Paseo Daemon | Paseo daemon HTTP (loopback helper also on 42013) |
|
||||
| 8080 | `*` | Ignis | Ignis browser-native Obsidian, see [[systems/ignis]] (Serve `/ignis`) |
|
||||
| 8085 | `0.0.0.0` | `agentmail-webhook.py` | AgentMail webhook receiver; funnel-exposed (`/webhook`) |
|
||||
| 8787 | `*` | `hermes-webui/server.py` | hermes-webui backend (Serve `/webui`); used as the health-probe target in [[runbooks/update-hermes-safely]] |
|
||||
| 9099 | `0.0.0.0` | `job_dashboard_server.py` | jobs dashboard + static routes (Serve `/jobs`) |
|
||||
| 9120 | `0.0.0.0` | `serve.py` (repo `public/`) | **Agent Estate Wiki site** (Serve `/wiki`) — added 2026-11-05 |
|
||||
| 5901 | `*` | x11vnc | VNC on Xvfb `:99`, see [[systems/headless-server]] |
|
||||
| 6080 / 6081 | `127.0.0.1` | websockify → 5900 / 5901 | noVNC bridges for the Camofox and desktop displays |
|
||||
| 25 | `127.0.0.1` / `[::1]` | postfix | local mail transport |
|
||||
|
||||
\* `curl` target used in [[runbooks/update-hermes-safely]] health check; binding not independently re-verified in this pass.
|
||||
Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode.
|
||||
|
||||
## Tailscale Serve / Funnel (verified 2026-11-05)
|
||||
|
||||
`tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public).
|
||||
|
||||
| Route | Target | Note |
|
||||
|---|---|---|
|
||||
| `/` | `127.0.0.1:9119` | Hermes Web UI |
|
||||
| `/ui` | `127.0.0.1:5175` | hermes-ui static build |
|
||||
| `/jobs` | `127.0.0.1:9099` | jobs dashboard |
|
||||
| `/wiki` | `100.118.5.51:9120` | Agent Estate Wiki |
|
||||
| `/webui` | `127.0.0.1:8787` | hermes-webui |
|
||||
| `/ignis` | `127.0.0.1:8080` | Ignis |
|
||||
| `/gitea` | `100.82.100.128:3000` | Gitea (ct207 tailnet IP) |
|
||||
| `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) |
|
||||
| `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC |
|
||||
| `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page |
|
||||
| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (2026-11-05)** |
|
||||
| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (2026-11-05)** |
|
||||
|
||||
- `:9443` (tailnet only) — `/` → `127.0.0.1:5173`.
|
||||
- Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver).
|
||||
|
||||
## Relevant runbooks
|
||||
- [[runbooks/diagnose-docker]]
|
||||
|
||||
Reference in New Issue
Block a user