stocktake 2026-10-14: SECOND Proxmox host m93p (100.104.51.98, PVE 9.2.0, unclustered, IDs collide) discovered — gitea/ntfy/freshrss/openweb were never on pve; tailnet inventory rebuilt 30 -> 45 nodes (15 missing); VM 282 running; Camofox recovered to 29.9M

skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages

- infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free)
- infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443)
- infra/network-map.md, current-state.md, index.md, log.md updated
- Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass
This commit is contained in:
Tony0410
2026-10-14 03:13:04 +08:00
parent 828913e23b
commit deb6a8eec2
8 changed files with 317 additions and 45 deletions
+11
View File
@@ -14,6 +14,17 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
# Current State
Updated: 2026-10-14
## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST)
- **⚠ A second, unclustered Proxmox hypervisor exists — `m93p` at `100.104.51.98`, PVE 9.2.0. DISCOVERED this pass.** `pvecm nodes` is empty on both hosts (no cluster), and guest IDs **collide**: `m93p` CT221 is gitea, `pve` CT103 is cliproxyapi, and both worlds share CT100/CT221/CT245 numbers. It had been in the Tailscale node list as an unremarkable `online` peer with no wiki page recording it as a hypervisor; a tailnet-wide `:8006` scan returned only `m93p` and `pve`, which is how it surfaced. Live guests on `m93p` include gitea (CT221), ntfy (CT1241), freshrss (CT1260), openweb (CT245), vaultwarden, obsidian-livesync, rustdesk, homebridge, litellm (CT500), omniroute (CT101), paseo (CT1261), mailflow, social-dashboard, docker-01, rclone, freellmapi; `nightbridge` stopped. → [[infrastructure/proxmox]], [[infrastructure/network-map]]
- **Four guests the wiki placed on `pve` are actually on `m93p`.** CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`,** yet all four services answer. `/etc/pve/lxc/` on `pve` holds only `260.conf.bak` and no vzdump log references them — so they were never deleted, **they were never on this host.** Every prior audit since 2026-10-11 carried the error forward. Always qualify a guest with its host. → [[infrastructure/proxmox]]
- **VM 282 `ubuntu-server-xfce` is running**, not stopped as recorded. → [[infrastructure/proxmox]]
- **Tailscale node inventory rebuilt: 45 non-funnel nodes** (plus 23 `funnel-ingress-node` IPv6 peers, excluded). The 2026-10-11 table had drifted badly — it listed 30 nodes and was **missing 15**, including `m93p`, `pve`, `omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`, `omarchybook`, `sftpgo`, `rclone`, `omni`, `runtipi`. Several states were wrong too (`anthonys-mac-mini` is online on a different IP; `omarchy-macmini`/`omarchy-quattro` are not live peers; `anthonys-iphone` is online not active). Serve routes on CT460 re-verified **exactly** as recorded — 13 routes, `:8443/webhook` Funnel, `:9443` tailnet-only. → [[infrastructure/tailscale]]
- **Mattermost CT255 confirmed stopped, consistent** with its Tailscale peer offline 24d — unchanged finding, re-verified. → [[infrastructure/proxmox]]
- **Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service` now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with 16 tasks, active since 2026-10-13 23:42:55. The containment limits held; the workload was the fault. → [[systems/browser-backend]]
- **Stocktake artifacts regenerated:** `systems/skills-index.md` = **346 skills / 70 categories** (the known 347-on-disk off-by-one persists); `raw/stocktake/freshness-20261014.txt` = **48 stale / 38 fresh / 7 no-date of 93 pages**, identical in shape to 10-13.
- **Quartz rebuild still inert** (no `quartz.config.ts`, no `npx`) — `public/` was not refreshed. Carried open item.
## Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST)
- **The four-day clean streak broke: 2026-10-13 = `167 completed` / 1 `failed`** (10-12 finalised `175 / 0`, 10-11 `172 / 0`, 10-10 `175 / 0`). The failure is `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) on `RuntimeError: HTTP 404: No active credentials for provider: opencode` — the **third** failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident `1a8e90_d11698bd7275` (`detected`, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → [[systems/scheduled-tasks]]
+6 -2
View File
@@ -1,6 +1,10 @@
# Anthony's Systems Wiki
> Operational knowledge for Anthony's agents, infrastructure,
> model routing and automation. Last updated: 2026-10-14
>
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
> Guest IDs collide between the two hosts.
## Start here
- [[current-state]] — Current systems status
@@ -64,8 +68,8 @@
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
### Networking
- [[infrastructure/proxmox]] — PVE host, VMs, LXC containers, storage datastores (`/mnt/ssd0` 93 % / ~63 GiB free, 2026-10-12)
- [[infrastructure/tailscale]] — Tailscale serve routes and VPN
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14)
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **45-node** tailnet inventory (2026-10-14)
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
## Concepts
+3 -2
View File
@@ -3,8 +3,8 @@ title: Network Map
type: infrastructure
status: active
created: 2026-07-22
updated: 2026-07-22
verified_on: 2026-07-22
updated: 2026-10-14
verified_on: 2026-10-14
confidence: high
tags: [infrastructure, network]
sources: [raw/configs/hermes-config-sanitized.txt]
@@ -23,6 +23,7 @@ Where Hermes and its dependencies run, and how traffic reaches them.
### Tailscale (`kangaroo-eel.ts.net`)
- **pve:** `100.96.100.82`
- **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host.
- **hermes (ct460):** `100.118.5.51`
- **hermesagent (ct450, the original/legacy Hermes agent dashboard):** `100.98.205.112`
- **omniroute (ct600):** resolves via `omniroute.kangaroo-eel.ts.net` → `100.88.81.19`
+86 -16
View File
@@ -3,8 +3,8 @@ title: Proxmox VE
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-10-12
verified_on: 2026-10-12
updated: 2026-10-14
verified_on: 2026-10-14
confidence: high
tags: [proxmox, virtualization, infrastructure, homelab]
---
@@ -21,7 +21,13 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
- **Primary user:** `Anthony@pam` (full admin)
- **Additional users:** `root@pam`, `glance@pve`, `prom-exporter@pve`, `homepage@pve`
## Live LXC Inventory (verified 2026-10-11 via `pct list`)
## Live LXC Inventory (verified 2026-10-14 via `pct list`)
> **Two Proxmox hosts exist — always qualify a guest with its host.** `pvecm nodes` is empty on
> both, so they are independent and unclustered. Guest IDs also collide — **CT221 (gitea) and
> CT245 (openweb) carry the same VMID on both hosts** — so an unqualified "CT221" is ambiguous.
> `m93p` (`100.104.51.98`, PVE 9.2.0) was **discovered 2026-10-14** and is documented in its own
> section below. It hosts several services the wiki had placed on `pve`.
| VMID | Name | Status | Purpose |
|---|---|---|---|
@@ -32,11 +38,7 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
| 207 | webserver | running | Web server |
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
| 215 | n8n | running | N8n automation |
| 221 | gitea | running | Gitea git server |
| 241 | ntfy | running | Ntfy push notifications |
| 245 | openweb | running | Open WebUI |
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer last seen 21d ago) |
| 260 | freshrss | running | FreshRSS feed reader |
| 270 | nextcloud | running | Nextcloud (apache2 active) |
| 299 | sftpgo | running | SFTPGo file server |
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
@@ -48,31 +50,99 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
| 600 | omniroute | **running** | OmniRoute — live server on `:20128` (see [[systems/omniroute]]) |
| 601 | omni | **stopped** | Retired successor to CT600 — stopped, `onboot=0`, offline 54d |
## Live QEMU Inventory (verified 2026-10-11 via `qm list`)
### ⚠ Four entries moved off this host (corrected 2026-10-14)
The 2026-10-11 inventory listed **CT221 (gitea), CT241 (ntfy), CT245 (openweb) and CT260
(freshrss)** as guests of `pve`. They are **absent from live `pct list`** on `pve` — yet all four
services are still up, because they live on `m93p`. `/etc/pve/lxc/` on `pve` holds only
`260.conf.bak` (no active config) and no vzdump task log references CT 221/241/245/260, so these
were never deleted here: **they were never on this host.** The 2026-10-11 audit carried the older
error forward.
| Wiki entry (removed) | Real host | Real VMID |
|---|---|---|
| ~~CT221 gitea~~ | `m93p` | **CT221** — same ID, different host (coincidence) |
| ~~CT241 ntfy~~ | `m93p` | **CT1241** |
| ~~CT245 openweb~~ | `m93p` | **CT245** — same ID, different host |
| ~~CT260 freshrss~~ | `m93p` | **CT1260** |
## Live QEMU Inventory (verified 2026-10-14 via `qm list`)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | homeassistant | running | Home Assistant (13GB RAM) |
| 110 | omarchy | running | Omarchy (13GB RAM, 106GB disk) — **new, not previously recorded** |
| 282 | ubuntu-server-xfce | **stopped** | Ubuntu desktop |
| 282 | ubuntu-server-xfce | **running** | Ubuntu desktop — recorded **stopped** until 2026-10-14; live `qm list` shows running |
| 310 | cachyos | **stopped** | CachyOS |
| 403 | open-claw | — | OpenClaw agent; **not present in `qm list`** — it runs elsewhere (Open-Claw VM, see [[infrastructure/hosts]]) |
| 401 | hermes-old | — | Legacy Hermes VM; **not present in `qm list`** — appears decommissioned/removed |
**Not in either list:** VM 403 (`open-claw`) and VM 401 (`hermes-old`) are referenced by the wiki but no longer appear in `qm list`. They have either been migrated off this host or deleted. Status unconfirmed — verify before relying on either entry.
## Storage (live, 2026-10-12 00:1x AWST — `df -h /mnt/ssd0` + `pvesm status`)
## Host `m93p` (100.104.51.98) — second Proxmox hypervisor, discovered 2026-10-14
Identified because a tailnet-wide scan for `:8006` returned only `m93p` and `pve`. It had been
sitting in the Tailscale node list as an unremarkable `online` peer, but no wiki page recorded it
as a hypervisor.
- **Hostname:** `m93p` · **Tailscale:** `100.104.51.98`
- **PVE version:** **9.2.0** (kernel `7.0.14-12-pve`, `pve-manager 9.2.10`) — materially newer than `pve`'s 8.x
- **Not clustered with `pve`:** `pvecm nodes` empty, no `/etc/pve/corosync.conf`
### LXC Inventory (verified 2026-10-14 via `pct list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 100 | obsidian-livesync | running | Obsidian LiveSync — ID does not collide; `pve` has no CT100 |
| 101 | omniroute | running | **An OmniRoute instance distinct from `pve`'s CT600/CT601 — see [[systems/omniroute]]; identity/role unconfirmed** |
| 102 | vaultwarden | running | Vaultwarden password manager |
| 104 | rclone | running | Rclone |
| 221 | gitea | running | **This is the host behind the wiki's `gitea` node — the Gitea remote lives here, not on `pve`** |
| 245 | openweb | running | Open WebUI — `pve` has no live CT245; the guest this page previously recorded on `pve` is this one |
| 500 | litellm | running | LiteLLM proxy |
| 998 | nightbridge | **stopped** | Nightbridge |
| 1001 | docker-01 | running | Secondary Docker host |
| 1006 | freellmapi | running | FreeLLM API |
| 1007 | homebridge | running | Homebridge |
| 1008 | rustdesk | running | RustDesk remote access |
| 1241 | ntfy | running | Ntfy push notifications — wiki's `CT241` |
| 1260 | freshrss | running | FreshRSS feed reader — wiki's `CT260` |
| 1261 | paseo | running | Paseo coding-agent daemon |
| 1262 | mailflow | running | MailFlow |
| 1263 | social-dashboard | running | Social dashboard |
### QEMU Inventory (verified 2026-10-14 via `qm list` on m93p)
| VMID | Name | Status | Notes |
|---|---|---|---|
| 103 | cachyos-m93p | **stopped** | CachyOS |
**Reachability note:** guest service ports on `m93p` were **not** reachable from CT460 during this
pass (they bind to their own interfaces and are not Tailscale-Served). Only hypervisor `:8006` on
`100.104.51.98` is open. Reach them via the host or a Tailscale Serve route before probing services.
**⚠ Follow-up:** `m93p`'s guest list is recorded here from a single `pct list` / `qm list` pass.
The pages [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]] and
[[infrastructure/docker-services]] still describe single-host topology and should be re-verified
against `m93p` — in particular whether `m93p`'s `omniroute` (CT101) is a fourth OmniRoute node or
a duplicate of CT600/CT601.
## Storage (live, 2026-10-14 03:0x AWST — `pvesm status` + `df -h /mnt/ssd0`, host `pve`)
| Datastore | Type | Status | Total | Used | Used % |
|---|---|---|---|---|---|
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **93 %** (`df`) / 88.1 % (`pvesm`) |
| `SSD1` | lvmthin | active | 931 GiB | 802 GiB | 86.1 % |
| `fred-qnap` | cifs | active | 13.4 TiB | 10.3 TiB | 76.8 % |
| `local` | dir | active | 67.7 GiB | 55.5 GiB | 82.0 % |
| `local-lvm` | lvmthin | active | 141 GiB | 19.2 GiB | 13.6 % |
| `ssd0` (`/mnt/ssd0`, `/dev/sdb`) | dir | active | 916 GiB | 808 GiB | **88.17 %** (`pvesm`) |
| `SSD1` | lvmthin | active | 976 GiB | 815 GiB | 87.47 % |
| `fred-qnap` | cifs | active | 13.7 TiB | 10.3 TiB | 77.22 % |
| `local` | dir | active | 67.7 GiB | 55.1 GiB | 81.68 % |
| `local-lvm` | lvmthin | active | 141 GiB | 19.0 GiB | 13.44 % |
| `iMacHDD` | cifs | **inactive** | — | — | — |
**`/mnt/ssd0` is the capacity pressure point: 93 % with only ~63 GiB free.** Both figures describe the same ~808 GiB used and differ only in denominator — `df` excludes reserved blocks (808/871 GiB), `pvesm` divides by the full 916 GiB (88.14 %). Treat the 93 % as the number that matters before adding a disk-heavy guest (VM 110 `omarchy` is ~106 GB on its own). `iMacHDD` is inactive (`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
**`ssd0` improved from the 93 % figure: now 88.17 % by `pvesm`, ~63 GiB free.** The earlier 93 %
came from `df`, which excludes reserved blocks; `pvesm` divides by the full 960 GiB. Both describe
the same ~808 GiB used. **Use the `pvesm` number when comparing across hosts** — it is the only
one that means the same thing on `m93p` too. `iMacHDD` remains inactive
(`storage 'iMacHDD' is not online`) — check the CIFS mount before using it as a target.
**Mount points on the host** (`ls -d /mnt/*`): `/mnt/ssd0`, `/mnt/pve`, `/mnt/vm`, `/mnt/rootfs`, plus per-guest mounts `/mnt/ct200`, `/mnt/vm110-esp-ro`, `/mnt/vm282`, `/mnt/vm283`, `/mnt/vm400`, `/mnt/vm402`, `/mnt/esp110`, `/mnt/test-nfs`, `/mnt/tmp` and retired ones (`/mnt/old-openclaw-ro`, `/mnt/openclaw-old`, `/mnt/vm400-backup`, `/mnt/vm400-export`, `/mnt/vm402-fix`).
+41 -23
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-10-11
verified_on: 2026-10-11
updated: 2026-10-14
verified_on: 2026-10-14
confidence: high
tags: [tailscale, vpn, networking, infrastructure]
---
@@ -19,42 +19,60 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-10-11)
Full node list from `tailscale status`. Offline peers are listed with their last-seen age.
## Tailscale Node Inventory (verified 2026-10-14)
Full node list from `tailscale status` — 45 non-funnel nodes. Offline peers are listed with their last-seen age.
Re-verified 2026-10-14: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
| Node | IP | State |
|---|---|---|
| hermes | 100.118.5.51 | **self** |
| agentzero | 100.106.54.94 | online |
| anthonys-imac | 100.112.123.124 | offline 17d |
| anthonys-imac | 100.112.123.124 | idle/offline 20d |
| anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | active |
| anthonys-mac-mini | 100.68.34.66 | offline 8d |
| cachyos | 100.124.208.15 | offline 37d |
| clawtest | 100.122.189.81 | online (CT333) |
| cliproxyapi | 100.83.112.94 | online (CT103) |
| cloudreve | 100.111.187.7 | offline 78d |
| conduit | 100.81.79.87 | offline 34d |
| docker | 100.96.150.69 | online (CT200) |
| anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.74.17.15 | online |
| cachyos | 100.124.208.15 | idle/offline 40d |
| clawtest | 100.122.189.81 | online |
| cliproxyapi | 100.83.112.94 | online |
| cloudreve | 100.111.187.7 | idle/offline 81d |
| conduit | 100.81.79.87 | idle/offline 37d |
| docker-01 | 100.121.12.31 | online |
| elementsynapse | 100.75.250.37 | offline 81d |
| docker | 100.96.150.69 | online |
| elementsynapse | 100.75.250.37 | idle/offline 84d |
| freellmapi | 100.127.208.75 | online |
| freshrss-new | 100.110.180.115 | idle/offline 57d |
| freshrss | 100.83.60.93 | online |
| freshrss-new | 100.110.180.115 | offline 54d |
| gitea | 100.82.100.128 | online (CT221) |
| hermesagent | 100.98.205.112 | online (CT450) |
| homeassistant | 100.68.139.5 | active, exit node |
| gitea | 100.82.100.128 | online |
| hermesagent | 100.98.205.112 | online |
| homeassistant | 100.68.139.5 | active; offers exit node |
| litellm | 100.79.29.64 | online |
| m93p | 100.104.51.98 | online |
| mailflow | 100.115.60.92 | online |
| mattermost | 100.98.116.14 | **idle/offline 21d** — CT255 is stopped, matches [[infrastructure/proxmox]] |
| mattermost | 100.98.116.14 | idle/offline 24d — CT255 stopped |
| n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online (CT270) |
| nextcloud | 100.90.102.19 | online |
| ntfy | 100.67.153.116 | active |
| ollama | 100.108.44.119 | online (CT205) |
| omarchy-macmini | 100.104.53.105 | offline 8d |
| omarchy-quattro | 100.106.27.27 | offline 40d |
| ollama | 100.108.44.119 | online |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | online |
| omni | 100.93.204.4 | idle/offline 57d |
| omniroute-1 | 100.78.63.20 | active |
| omniroute | 100.88.81.19 | online |
| openweb | 100.69.230.13 | online |
| paseo | 100.72.188.93 | online |
| pve | 100.96.100.82 | online |
| qwenpaw | 100.110.72.126 | online |
| rclone | 100.107.116.17 | online |
| runtipi | 100.96.244.39 | online |
| rustdesk | 100.95.227.57 | online |
| sftpgo | 100.88.223.70 | active |
| social-dashboard | 100.70.183.37 | online |
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 53d |
| vaultwarden | 100.123.225.90 | online |
| webserver | 100.82.40.23 | online |
> Note: `tailscale status` also lists 23 `funnel-ingress-node` peers with IPv6-only addresses — these are Tailscale's Funnel infrastructure, not homelab hosts, and are deliberately excluded above.
>
> Note: Tailscale's offline list is not authoritative for container state — cross-check `pct list` on the PVE host.
## CT460 Serve Routes
+62
View File
@@ -653,3 +653,65 @@ session.
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md`
(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt`
(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again
(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item.
**Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0
(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and
guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale
node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a
hypervisor. Live guests recorded from a single `pct list` / `qm list` pass.
**Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**,
yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak`
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never
on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward.
**Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live
`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now
noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`,
`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`,
`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too
(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not
live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified
**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that
table needed no change.
**Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service`
now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
fault, as diagnosed last cycle.
**Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282
`ubuntu-server-xfce` is now **running**, not stopped as recorded.
**Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed /
1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23
completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open
incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06`
(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot
Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled /
3 disabled**.
**Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry
correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer
exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter),
`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header
warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt`
(new), `log.md`.
**Not verified, left as stale:** the remaining 46 stale pages were not probed against a live
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]],
[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
+106
View File
@@ -0,0 +1,106 @@
# Wiki freshness audit — 2026-10-14
Threshold: >21 days since `updated`
## STALE (48)
- [ 84d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
- [ 84d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
- [ 84d] concepts/agent-memory.md (updated 2026-07-22)
- [ 84d] concepts/model-routing.md (updated 2026-07-22)
- [ 84d] concepts/persistent-shells.md (updated 2026-07-22)
- [ 84d] concepts/tool-calling.md (updated 2026-07-22)
- [ 84d] decisions/index.md (updated 2026-07-22)
- [ 84d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
- [ 84d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
- [ 84d] experiments/index.md (updated 2026-07-22)
- [ 84d] infrastructure/backups.md (updated 2026-07-22)
- [ 84d] infrastructure/credentials-map.md (updated 2026-07-22)
- [ 84d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
- [ 84d] infrastructure/network-map.md (updated 2026-07-22)
- [ 84d] runbooks/backup-wiki.md (updated 2026-07-22)
- [ 84d] runbooks/diagnose-docker.md (updated 2026-07-22)
- [ 84d] runbooks/recover-docker-service.md (updated 2026-07-22)
- [ 84d] runbooks/restart-browser.md (updated 2026-07-22)
- [ 84d] runbooks/restart-hermes.md (updated 2026-07-22)
- [ 84d] runbooks/restore-openrouter.md (updated 2026-07-22)
- [ 84d] runbooks/test-openrouter.md (updated 2026-07-22)
- [ 84d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
- [ 84d] systems/cloudflare-tunnel.md (updated 2026-07-22)
- [ 84d] systems/headless-server.md (updated 2026-07-22)
- [ 84d] systems/openrouter.md (updated 2026-07-22)
- [ 84d] systems/terminal-backend.md (updated 2026-07-22)
- [ 81d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
- [ 81d] systems/tool-search.md (updated 2026-07-25)
- [ 80d] systems/homepage-dashboard.md (updated 2026-07-26)
- [ 80d] systems/obsidian.md (updated 2026-07-26)
- [ 78d] runbooks/multiplexer-setup.md (updated 2026-07-28)
- [ 77d] systems/opencode-go.md (updated 2026-07-29)
- [ 76d] systems/hermes-ui.md (updated 2026-07-30)
- [ 74d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
- [ 73d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
- [ 73d] reports/uber-eats/index.md (updated 2026-08-02)
- [ 63d] comparisons/memory-backends.md (updated 2026-08-12)
- [ 63d] comparisons/model-routers.md (updated 2026-08-12)
- [ 61d] systems/guanaco.md (updated 2026-08-14)
- [ 60d] agents/index.md (updated 2026-08-15)
- [ 60d] agents/open-claw.md (updated 2026-08-15)
- [ 60d] agents/zeroclaw.md (updated 2026-08-15)
- [ 60d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
- [ 59d] CONTRIBUTING.md (updated 2026-08-16)
- [ 58d] systems/nextcloud.md (updated 2026-08-17)
- [ 47d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
- [ 46d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
- [ 40d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
## FRESH (38)
- [ 10d] agents/hermes-chaos.md
- [ 10d] agents/hermes-production.md
- [ 3d] agents/nanobot.md
- [ 3d] agents/qwenpaw.md
- [ 1d] comparisons/steel-vs-camoufox.md
- [ 1d] concepts/browser-automation.md
- [ 0d] current-state.md
- [ 2d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 11d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 3d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 8d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 10d] incidents/2026-10-03-ct450-hermes-update.md
- [ 10d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 8d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 1d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 5d] incidents/2026-10-09-memory-recall-relevance.md
- [ 3d] infrastructure/docker-services.md
- [ 10d] infrastructure/hosts.md
- [ 2d] infrastructure/proxmox.md
- [ 3d] infrastructure/tailscale.md
- [ 2d] queries/retained-research-answers.md
- [ 1d] runbooks/job-radar.md
- [ 10d] runbooks/provider-health.md
- [ 10d] runbooks/update-hermes-safely.md
- [ 1d] systems/browser-backend.md
- [ 12d] systems/current-profile.md
- [ 3d] systems/hermes-agent.md
- [ 2d] systems/home-assistant.md
- [ 3d] systems/ignis.md
- [ 10d] systems/memory-backend.md
- [ 3d] systems/messaging-integrations.md
- [ 1d] systems/mnemosyne.md
- [ 13d] systems/model-providers.md
- [ 2d] systems/nanobot.md
- [ 3d] systems/omniroute.md
- [ 3d] systems/paseo.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
## NO updated DATE (7)
- SCHEMA.md
- experiments/active/example.md
- incidents/2026-07-29-mnemosyne-relay.md
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- index.md
- log.md
- reports/uber-eats/2026-07.md
TOTAL: 48 stale, 38 fresh, 7 no-date of 93 pages
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system
status: active
created: 2026-07-22
updated: 2026-10-13
verified_on: 2026-10-13
updated: 2026-10-14
verified_on: 2026-10-14
confidence: high
tags: [skills, hermes]
sources: []