stocktake 2026-10-14: SECOND Proxmox host m93p (100.104.51.98, PVE 9.2.0, unclustered, IDs collide) discovered — gitea/ntfy/freshrss/openweb were never on pve; tailnet inventory rebuilt 30 -> 45 nodes (15 missing); VM 282 running; Camofox recovered to 29.9M
skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages - infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free) - infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443) - infra/network-map.md, current-state.md, index.md, log.md updated - Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass
This commit is contained in:
@@ -653,3 +653,65 @@ session.
|
||||
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
|
||||
|
||||
No secrets written; job/incident IDs, paths and tailnet addresses only.
|
||||
|
||||
## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
|
||||
|
||||
Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md`
|
||||
(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt`
|
||||
(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again
|
||||
(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item.
|
||||
|
||||
**Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0
|
||||
(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and
|
||||
guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and
|
||||
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
|
||||
hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale
|
||||
node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a
|
||||
hypervisor. Live guests recorded from a single `pct list` / `qm list` pass.
|
||||
|
||||
**Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241
|
||||
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**,
|
||||
yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak`
|
||||
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never
|
||||
on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward.
|
||||
|
||||
**Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live
|
||||
`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now
|
||||
noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`,
|
||||
`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`,
|
||||
`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too
|
||||
(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not
|
||||
live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified
|
||||
**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that
|
||||
table needed no change.
|
||||
|
||||
**Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service`
|
||||
now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with
|
||||
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
|
||||
fault, as diagnosed last cycle.
|
||||
|
||||
**Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282
|
||||
`ubuntu-server-xfce` is now **running**, not stopped as recorded.
|
||||
|
||||
**Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed /
|
||||
1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23
|
||||
completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open
|
||||
incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06`
|
||||
(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot
|
||||
Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled /
|
||||
3 disabled**.
|
||||
|
||||
**Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry
|
||||
correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer
|
||||
exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter),
|
||||
`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header
|
||||
warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt`
|
||||
(new), `log.md`.
|
||||
|
||||
**Not verified, left as stale:** the remaining 46 stale pages were not probed against a live
|
||||
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
|
||||
single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]],
|
||||
[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the
|
||||
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
|
||||
|
||||
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
|
||||
|
||||
Reference in New Issue
Block a user