Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
When: 2026-09-02, ~01:45 AWST
Duration: ~30 minutes
Affected: SFTPGo service on CT299 (LXC 299), dashboard monitoring
Root cause: CT299's tailscaled died (likely OOM — CT299 swap was 100% full at 43Gi used). When attempting to restart the container, PVE's pre-start hook failed because 5 stale lxc-attach processes from earlier diagnostic probes had pinned CT299's cgroup.
Timeline
Time (AWST)
Event
~01:45
Dashboard reports SFTPGo service down
01:45–02:00
Investigation: CT299 unreachable on Tailscale (offline, last seen 7h ago), all TCP ports timeout
02:00–02:10
Root cause identified: CT299 swap exhausted (43Gi used, 100%), tailscaled inside CT299 had died; PVE pct start 299 fails with pre-start hook error: failed to remove directory /sys/fs/cgroup/lxc/299/ns/.lxc: Device or resource busy
02:10–02:15
Stale lxc-attach PIDs identified: 3037951, 3334928, 3352203, 3402511, 4004148 — killed all five
Live state checked: CT299 sftpgo is running on pve (100.96.100.82) per pct list;
the node is active in tailscale status at 100.88.223.70; the SFTPGo health endpoint
http://100.88.223.70:10080/healthz returns HTTP 401 (expected — auth required, service up).
The 2026-09-02 outage remains resolved; no recurrence.
Lessons Learned
Clean up lxc-attach probes — always exit cleanly and verify no lingering processes after diagnostic sessions
CT299 swap over-provisioned — 43Gi swap on 31Gi RAM is excessive; consider reducing swap allocation or trimming workload
Dashboard correctly flagged the outage — even though root cause was inside the LXC (tailscaled died), the dashboard's reachability check caught it