From deb6a8eec223eb360bd03fbcd42cb224deed3d7c Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Wed, 14 Oct 2026 03:13:04 +0800 Subject: [PATCH] =?UTF-8?q?stocktake=202026-10-14:=20SECOND=20Proxmox=20ho?= =?UTF-8?q?st=20m93p=20(100.104.51.98,=20PVE=209.2.0,=20unclustered,=20IDs?= =?UTF-8?q?=20collide)=20discovered=20=E2=80=94=20gitea/ntfy/freshrss/open?= =?UTF-8?q?web=20were=20never=20on=20pve;=20tailnet=20inventory=20rebuilt?= =?UTF-8?q?=2030=20->=2045=20nodes=20(15=20missing);=20VM=20282=20running;?= =?UTF-8?q?=20Camofox=20recovered=20to=2029.9M?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages - infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free) - infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443) - infra/network-map.md, current-state.md, index.md, log.md updated - Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass --- current-state.md | 11 +++ index.md | 8 +- infrastructure/network-map.md | 5 +- infrastructure/proxmox.md | 102 ++++++++++++++++++++++---- infrastructure/tailscale.md | 64 ++++++++++------ log.md | 62 ++++++++++++++++ raw/stocktake/freshness-20261014.txt | 106 +++++++++++++++++++++++++++ systems/skills-index.md | 4 +- 8 files changed, 317 insertions(+), 45 deletions(-) create mode 100644 raw/stocktake/freshness-20261014.txt diff --git a/current-state.md b/current-state.md index 8574bd8..fce9e0f 100644 --- a/current-state.md +++ b/current-state.md @@ -14,6 +14,17 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv # Current State Updated: 2026-10-14 +## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST) + +- **⚠ A second, unclustered Proxmox hypervisor exists — `m93p` at `100.104.51.98`, PVE 9.2.0. DISCOVERED this pass.** `pvecm nodes` is empty on both hosts (no cluster), and guest IDs **collide**: `m93p` CT221 is gitea, `pve` CT103 is cliproxyapi, and both worlds share CT100/CT221/CT245 numbers. It had been in the Tailscale node list as an unremarkable `online` peer with no wiki page recording it as a hypervisor; a tailnet-wide `:8006` scan returned only `m93p` and `pve`, which is how it surfaced. Live guests on `m93p` include gitea (CT221), ntfy (CT1241), freshrss (CT1260), openweb (CT245), vaultwarden, obsidian-livesync, rustdesk, homebridge, litellm (CT500), omniroute (CT101), paseo (CT1261), mailflow, social-dashboard, docker-01, rclone, freellmapi; `nightbridge` stopped. → [[infrastructure/proxmox]], [[infrastructure/network-map]] +- **Four guests the wiki placed on `pve` are actually on `m93p`.** CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`,** yet all four services answer. `/etc/pve/lxc/` on `pve` holds only `260.conf.bak` and no vzdump log references them — so they were never deleted, **they were never on this host.** Every prior audit since 2026-10-11 carried the error forward. Always qualify a guest with its host. → [[infrastructure/proxmox]] +- **VM 282 `ubuntu-server-xfce` is running**, not stopped as recorded. → [[infrastructure/proxmox]] +- **Tailscale node inventory rebuilt: 45 non-funnel nodes** (plus 23 `funnel-ingress-node` IPv6 peers, excluded). The 2026-10-11 table had drifted badly — it listed 30 nodes and was **missing 15**, including `m93p`, `pve`, `omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`, `omarchybook`, `sftpgo`, `rclone`, `omni`, `runtipi`. Several states were wrong too (`anthonys-mac-mini` is online on a different IP; `omarchy-macmini`/`omarchy-quattro` are not live peers; `anthonys-iphone` is online not active). Serve routes on CT460 re-verified **exactly** as recorded — 13 routes, `:8443/webhook` Funnel, `:9443` tailnet-only. → [[infrastructure/tailscale]] +- **Mattermost CT255 confirmed stopped, consistent** with its Tailscale peer offline 24d — unchanged finding, re-verified. → [[infrastructure/proxmox]] +- **Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service` now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with 16 tasks, active since 2026-10-13 23:42:55. The containment limits held; the workload was the fault. → [[systems/browser-backend]] +- **Stocktake artifacts regenerated:** `systems/skills-index.md` = **346 skills / 70 categories** (the known 347-on-disk off-by-one persists); `raw/stocktake/freshness-20261014.txt` = **48 stale / 38 fresh / 7 no-date of 93 pages**, identical in shape to 10-13. +- **Quartz rebuild still inert** (no `quartz.config.ts`, no `npx`) — `public/` was not refreshed. Carried open item. + ## Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST) - **The four-day clean streak broke: 2026-10-13 = `167 completed` / 1 `failed`** (10-12 finalised `175 / 0`, 10-11 `172 / 0`, 10-10 `175 / 0`). The failure is `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) on `RuntimeError: HTTP 404: No active credentials for provider: opencode` — the **third** failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident `1a8e90_d11698bd7275` (`detected`, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → [[systems/scheduled-tasks]] diff --git a/index.md b/index.md index c844ecc..186b41d 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,10 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, > model routing and automation. Last updated: 2026-10-14 +> +> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered +> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. +> Guest IDs collide between the two hosts. ## Start here - [[current-state]] — Current systems status @@ -64,8 +68,8 @@ - [[infrastructure/backups]] — hermes backup CLI, retention, and gaps ### Networking -- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers, storage datastores (`/mnt/ssd0` 93 % / ~63 GiB free, 2026-10-12) -- [[infrastructure/tailscale]] — Tailscale serve routes and VPN +- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14) +- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **45-node** tailnet inventory (2026-10-14) - [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly ## Concepts diff --git a/infrastructure/network-map.md b/infrastructure/network-map.md index 2634665..28cc54e 100644 --- a/infrastructure/network-map.md +++ b/infrastructure/network-map.md @@ -3,8 +3,8 @@ title: Network Map type: infrastructure status: active created: 2026-07-22 -updated: 2026-07-22 -verified_on: 2026-07-22 +updated: 2026-10-14 +verified_on: 2026-10-14 confidence: high tags: [infrastructure, network] sources: [raw/configs/hermes-config-sanitized.txt] @@ -23,6 +23,7 @@ Where Hermes and its dependencies run, and how traffic reaches them. ### Tailscale (`kangaroo-eel.ts.net`) - **pve:** `100.96.100.82` +- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host. - **hermes (ct460):** `100.118.5.51` - **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112` - **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19` diff --git a/infrastructure/proxmox.md b/infrastructure/proxmox.md index 3ca2059..cbfe5fe 100644 --- a/infrastructure/proxmox.md +++ b/infrastructure/proxmox.md @@ -3,8 +3,8 @@ title: Proxmox VE type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-12 -verified_on: 2026-10-12 +updated: 2026-10-14 +verified_on: 2026-10-14 confidence: high tags: [proxmox, virtualization, infrastructure, homelab] --- @@ -21,7 +21,13 @@ Hypervisor hosting all VMs and LXC containers for the homelab. - **Primary user:** `Anthony@pam` (full admin) - **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve` -## Live LXC Inventory (verified 2026-10-11 via `pct list`) +## Live LXC Inventory (verified 2026-10-14 via `pct list`) + +> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on +> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and +> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous. +> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own +> section below. It hosts several services the wiki had placed on `pve`. | VMID | Name | Status | Purpose | |---|---|---|---| @@ -32,11 +38,7 @@ Hypervisor hosting all VMs and LXC containers for the homelab. | 207 | webserver | running | Web server | | 209 | cloudreve | **stopped** | Cloudreve (deprecated) | | 215 | n8n | running | N8n automation | -| 221 | gitea | running | Gitea git server | -| 241 | ntfy | running | Ntfy push notifications | -| 245 | openweb | running | Open WebUI | | 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer last seen 21d ago) | -| 260 | freshrss | running | FreshRSS feed reader | | 270 | nextcloud | running | Nextcloud (apache2 active) | | 299 | sftpgo | running | SFTPGo file server | | 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw | @@ -48,31 +50,99 @@ Hypervisor hosting all VMs and LXC containers for the homelab. | 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) | | 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d | -## Live QEMU Inventory (verified 2026-10-11 via `qm list`) +### ⚠ Four entries moved off this host (corrected 2026-10-14) + +The 2026-10-11 inventory listed **CT221 (gitea), CT241 (ntfy), CT245 (openweb) and CT260 +(freshrss)** as guests of `pve`. They are **absent from live `pct list`** on `pve` — yet all four +services are still up, because they live on `m93p`. `/etc/pve/lxc/` on `pve` holds only +`260.conf.bak` (no active config) and no vzdump task log references CT 221/241/245/260, so these +were never deleted here: **they were never on this host.** The 2026-10-11 audit carried the older +error forward. + +| Wiki entry (removed) | Real host | Real VMID | +|---|---|---| +| ~~CT221 gitea~~ | `m93p` | **CT221** — same ID, different host (coincidence) | +| ~~CT241 ntfy~~ | `m93p` | **CT1241** | +| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host | +| ~~CT260 freshrss~~ | `m93p` | **CT1260** | + +## Live QEMU Inventory (verified 2026-10-14 via `qm list`) | VMID | Name | Status | Notes | |---|---|---|---| | 100 | homeassistant | running | Home Assistant (13GB RAM) | | 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** | -| 282 | ubuntu-server-xfce | **stopped** | Ubuntu desktop | +| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running | | 310 | cachyos | **stopped** | CachyOS | | 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) | | 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed | **Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry. -## Storage (live, 2026-10-12 00:1x AWST — `df -h /mnt/ssd0` + `pvesm status`) +## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14 + +Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been +sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki page recorded it +as a hypervisor. + +- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98` +- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x +- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf` + +### LXC Inventory (verified 2026-10-14 via `pct list` on m93p) + +| VMID | Name | Status | Notes | +|---|---|---|---| +| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 | +| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** | +| 102 | vaultwarden | running | Vaultwarden password manager | +| 104 | rclone | running | Rclone | +| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** | +| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one | +| 500 | litellm | running | LiteLLM proxy | +| 998 | nightbridge | **stopped** | Nightbridge | +| 1001 | docker-01 | running | Secondary Docker host | +| 1006 | freellmapi | running | FreeLLM API | +| 1007 | homebridge | running | Homebridge | +| 1008 | rustdesk | running | RustDesk remote access | +| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` | +| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` | +| 1261 | paseo | running | Paseo coding-agent daemon | +| 1262 | mailflow | running | MailFlow | +| 1263 | social-dashboard | running | Social dashboard | + +### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p) + +| VMID | Name | Status | Notes | +|---|---|---|---| +| 103 | cachyos-m93p | **stopped** | CachyOS | + +**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this +pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on +`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services. + +**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass. +The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and +[[infrastructure/docker-services]] still describe single-host topology and should be re-verified +against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or +a duplicate of CT600/CT601. + +## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`) | Datastore | Type | Status | Total | Used | Used % | |---|---|---|---|---|---| -| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **93 %** (`df`) / 88.1 % (`pvesm`) | -| `SSD1` | lvmthin | active | 931 GiB | 802 GiB | 86.1 % | -| `fred-qnap` | cifs | active | 13.4 TiB | 10.3 TiB | 76.8 % | -| `local` | dir | active | 67.7 GiB | 55.5 GiB | 82.0 % | -| `local-lvm` | lvmthin | active | 141 GiB | 19.2 GiB | 13.6 % | +| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) | +| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % | +| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % | +| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % | +| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % | | `iMacHDD` | cifs | **inactive** | — | — | — | -**`/mnt/ssd0` is the capacity pressure point: 93 % with only ~63 GiB free.** Both figures describe the same ~808 GiB used and differ only in denominator — `df` excludes reserved blocks (808/871 GiB), `pvesm` divides by the full 916 GiB (88.14 %). Treat the 93 % as the number that matters before adding a disk-heavy guest (VM 110 `omarchy` is ~106 GB on its own). `iMacHDD` is inactive (`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target. +**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 % +came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe +the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only +one that means the same thing on `m93p` too. `iMacHDD` remains inactive +(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target. **Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`). diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 0f0666d..265ada7 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-11 -verified_on: 2026-10-11 +updated: 2026-10-14 +verified_on: 2026-10-14 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,42 +19,60 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-10-11) -Full node list from `tailscale status`. Offline peers are listed with their last-seen age. +## Tailscale Node Inventory (verified 2026-10-14) +Full node list from `tailscale status` — 45 non-funnel nodes. Offline peers are listed with their last-seen age. +Re-verified 2026-10-14: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). | Node | IP | State | |---|---|---| | hermes | 100.118.5.51 | **self** | | agentzero | 100.106.54.94 | online | -| anthonys-imac | 100.112.123.124 | offline 17d | +| anthonys-imac | 100.112.123.124 | idle/offline 20d | | anthonys-ipad | 100.111.198.120 | online | -| anthonys-iphone | 100.100.116.67 | active | -| anthonys-mac-mini | 100.68.34.66 | offline 8d | -| cachyos | 100.124.208.15 | offline 37d | -| clawtest | 100.122.189.81 | online (CT333) | -| cliproxyapi | 100.83.112.94 | online (CT103) | -| cloudreve | 100.111.187.7 | offline 78d | -| conduit | 100.81.79.87 | offline 34d | -| docker | 100.96.150.69 | online (CT200) | +| anthonys-iphone | 100.100.116.67 | online | +| anthonys-mac-mini | 100.74.17.15 | online | +| cachyos | 100.124.208.15 | idle/offline 40d | +| clawtest | 100.122.189.81 | online | +| cliproxyapi | 100.83.112.94 | online | +| cloudreve | 100.111.187.7 | idle/offline 81d | +| conduit | 100.81.79.87 | idle/offline 37d | | docker-01 | 100.121.12.31 | online | -| elementsynapse | 100.75.250.37 | offline 81d | +| docker | 100.96.150.69 | online | +| elementsynapse | 100.75.250.37 | idle/offline 84d | | freellmapi | 100.127.208.75 | online | +| freshrss-new | 100.110.180.115 | idle/offline 57d | | freshrss | 100.83.60.93 | online | -| freshrss-new | 100.110.180.115 | offline 54d | -| gitea | 100.82.100.128 | online (CT221) | -| hermesagent | 100.98.205.112 | online (CT450) | -| homeassistant | 100.68.139.5 | active, exit node | +| gitea | 100.82.100.128 | online | +| hermesagent | 100.98.205.112 | online | +| homeassistant | 100.68.139.5 | active; offers exit node | | litellm | 100.79.29.64 | online | | m93p | 100.104.51.98 | online | | mailflow | 100.115.60.92 | online | -| mattermost | 100.98.116.14 | **idle/offline 21d** — CT255 is stopped, matches [[infrastructure/proxmox]] | +| mattermost | 100.98.116.14 | idle/offline 24d — CT255 stopped | | n8n | 100.84.124.30 | online | -| nextcloud | 100.90.102.19 | online (CT270) | +| nextcloud | 100.90.102.19 | online | | ntfy | 100.67.153.116 | active | -| ollama | 100.108.44.119 | online (CT205) | -| omarchy-macmini | 100.104.53.105 | offline 8d | -| omarchy-quattro | 100.106.27.27 | offline 40d | +| ollama | 100.108.44.119 | online | +| omarchy | 100.73.227.103 | online | +| omarchybook | 100.107.9.100 | online | +| omni | 100.93.204.4 | idle/offline 57d | +| omniroute-1 | 100.78.63.20 | active | +| omniroute | 100.88.81.19 | online | +| openweb | 100.69.230.13 | online | +| paseo | 100.72.188.93 | online | +| pve | 100.96.100.82 | online | +| qwenpaw | 100.110.72.126 | online | +| rclone | 100.107.116.17 | online | +| runtipi | 100.96.244.39 | online | +| rustdesk | 100.95.227.57 | online | +| sftpgo | 100.88.223.70 | active | +| social-dashboard | 100.70.183.37 | online | +| ubuntu-server-xfce | 100.101.26.104 | idle/offline 53d | +| vaultwarden | 100.123.225.90 | online | +| webserver | 100.82.40.23 | online | +> Note: `tailscale status` also lists 23 `funnel-ingress-node` peers with IPv6-only addresses — these are Tailscale's Funnel infrastructure, not homelab hosts, and are deliberately excluded above. +> > Note: Tailscale's offline list is not authoritative for container state — cross-check `pct list` on the PVE host. ## CT460 Serve Routes diff --git a/log.md b/log.md index c95a2a2..97479a6 100644 --- a/log.md +++ b/log.md @@ -653,3 +653,65 @@ session. copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report. No secrets written; job/incident IDs, paths and tailnet addresses only. + +## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt + +Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md` +(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt` +(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again +(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item. + +**Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0 +(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and +guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and +CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the +hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale +node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a +hypervisor. Live guests recorded from a single `pct list` / `qm list` pass. + +**Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241 +(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**, +yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak` +and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never +on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward. + +**Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live +`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now +noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`, +`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`, +`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too +(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not +live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified +**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that +table needed no change. + +**Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service` +now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with +16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the +fault, as diagnosed last cycle. + +**Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282 +`ubuntu-server-xfce` is now **running**, not stopped as recorded. + +**Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed / +1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23 +completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open +incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06` +(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot +Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled / +3 disabled**. + +**Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry +correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer +exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter), +`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header +warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt` +(new), `log.md`. + +**Not verified, left as stale:** the remaining 46 stale pages were not probed against a live +source, so they stay flagged rather than rewritten. Not re-verified this pass and still +single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]], +[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the +OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story. + +No secrets written; host names, IPs, VMIDs and job/incident IDs only. diff --git a/raw/stocktake/freshness-20261014.txt b/raw/stocktake/freshness-20261014.txt new file mode 100644 index 0000000..b4a8971 --- /dev/null +++ b/raw/stocktake/freshness-20261014.txt @@ -0,0 +1,106 @@ +# Wiki freshness audit — 2026-10-14 +Threshold: >21 days since `updated` + +## STALE (48) + +- [ 84d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22) +- [ 84d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22) +- [ 84d] concepts/agent-memory.md (updated 2026-07-22) +- [ 84d] concepts/model-routing.md (updated 2026-07-22) +- [ 84d] concepts/persistent-shells.md (updated 2026-07-22) +- [ 84d] concepts/tool-calling.md (updated 2026-07-22) +- [ 84d] decisions/index.md (updated 2026-07-22) +- [ 84d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22) +- [ 84d] decisions/persistent-shell-enabled.md (updated 2026-07-22) +- [ 84d] experiments/index.md (updated 2026-07-22) +- [ 84d] infrastructure/backups.md (updated 2026-07-22) +- [ 84d] infrastructure/credentials-map.md (updated 2026-07-22) +- [ 84d] infrastructure/domains-and-tunnels.md (updated 2026-07-22) +- [ 84d] infrastructure/network-map.md (updated 2026-07-22) +- [ 84d] runbooks/backup-wiki.md (updated 2026-07-22) +- [ 84d] runbooks/diagnose-docker.md (updated 2026-07-22) +- [ 84d] runbooks/recover-docker-service.md (updated 2026-07-22) +- [ 84d] runbooks/restart-browser.md (updated 2026-07-22) +- [ 84d] runbooks/restart-hermes.md (updated 2026-07-22) +- [ 84d] runbooks/restore-openrouter.md (updated 2026-07-22) +- [ 84d] runbooks/test-openrouter.md (updated 2026-07-22) +- [ 84d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22) +- [ 84d] systems/cloudflare-tunnel.md (updated 2026-07-22) +- [ 84d] systems/headless-server.md (updated 2026-07-22) +- [ 84d] systems/openrouter.md (updated 2026-07-22) +- [ 84d] systems/terminal-backend.md (updated 2026-07-22) +- [ 81d] runbooks/gateway-resource-resilience.md (updated 2026-07-25) +- [ 81d] systems/tool-search.md (updated 2026-07-25) +- [ 80d] systems/homepage-dashboard.md (updated 2026-07-26) +- [ 80d] systems/obsidian.md (updated 2026-07-26) +- [ 78d] runbooks/multiplexer-setup.md (updated 2026-07-28) +- [ 77d] systems/opencode-go.md (updated 2026-07-29) +- [ 76d] systems/hermes-ui.md (updated 2026-07-30) +- [ 74d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01) +- [ 73d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02) +- [ 73d] reports/uber-eats/index.md (updated 2026-08-02) +- [ 63d] comparisons/memory-backends.md (updated 2026-08-12) +- [ 63d] comparisons/model-routers.md (updated 2026-08-12) +- [ 61d] systems/guanaco.md (updated 2026-08-14) +- [ 60d] agents/index.md (updated 2026-08-15) +- [ 60d] agents/open-claw.md (updated 2026-08-15) +- [ 60d] agents/zeroclaw.md (updated 2026-08-15) +- [ 60d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15) +- [ 59d] CONTRIBUTING.md (updated 2026-08-16) +- [ 58d] systems/nextcloud.md (updated 2026-08-17) +- [ 47d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28) +- [ 46d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29) +- [ 40d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04) + +## FRESH (38) + +- [ 10d] agents/hermes-chaos.md +- [ 10d] agents/hermes-production.md +- [ 3d] agents/nanobot.md +- [ 3d] agents/qwenpaw.md +- [ 1d] comparisons/steel-vs-camoufox.md +- [ 1d] concepts/browser-automation.md +- [ 0d] current-state.md +- [ 2d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 11d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 3d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 8d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 10d] incidents/2026-10-03-ct450-hermes-update.md +- [ 10d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 8d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 1d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 5d] incidents/2026-10-09-memory-recall-relevance.md +- [ 3d] infrastructure/docker-services.md +- [ 10d] infrastructure/hosts.md +- [ 2d] infrastructure/proxmox.md +- [ 3d] infrastructure/tailscale.md +- [ 2d] queries/retained-research-answers.md +- [ 1d] runbooks/job-radar.md +- [ 10d] runbooks/provider-health.md +- [ 10d] runbooks/update-hermes-safely.md +- [ 1d] systems/browser-backend.md +- [ 12d] systems/current-profile.md +- [ 3d] systems/hermes-agent.md +- [ 2d] systems/home-assistant.md +- [ 3d] systems/ignis.md +- [ 10d] systems/memory-backend.md +- [ 3d] systems/messaging-integrations.md +- [ 1d] systems/mnemosyne.md +- [ 13d] systems/model-providers.md +- [ 2d] systems/nanobot.md +- [ 3d] systems/omniroute.md +- [ 3d] systems/paseo.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md + +## NO updated DATE (7) + +- SCHEMA.md +- experiments/active/example.md +- incidents/2026-07-29-mnemosyne-relay.md +- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- index.md +- log.md +- reports/uber-eats/2026-07.md + +TOTAL: 48 stale, 38 fresh, 7 no-date of 93 pages \ No newline at end of file diff --git a/systems/skills-index.md b/systems/skills-index.md index 661d32e..1a62d48 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-10-13 -verified_on: 2026-10-13 +updated: 2026-10-14 +verified_on: 2026-10-14 confidence: high tags: [skills, hermes] sources: []