Commit Graph
35 Commits
Author SHA1 Message Date
Tony0410 38e96375fc audit 2026-10-23: current-state 2026-10-23 block; log entry 2026-10-23 00:18:26 +08:00
Tony0410 59d4b2e695 audit 2026-10-23: scheduled-tasks window (172/0, 5 incidents closed); reopen skills-index drift; skills-index generator now excludes .archive (306 skills, was 347) 2026-10-23 00:14:49 +08:00
Tony0410 1d92de31a0 🦉 wiki maintenance 2026-10-22: Quartz rebuild+deploy (new incident page live), skills-index 348, freshness 1 stale/95 fresh 2026-10-22 00:19:03 +08:00
Tony0410 c1d3b59260 🦉 wiki stocktake 2026-10-21: tailscale node table rebuilt, 2 missing Uber Eats reports recovered
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).

infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
  The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
  (100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
  but absent from the table body, omarchymac now offline, all offline ages stale.
  Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
  the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
  collides silently. Now 51 non-funnel nodes. Container states cross-checked against
  live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.

reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
  ~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
  missing from the published wiki. Root cause fixed in
  ~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
  ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
  and now emits YAML frontmatter (without it the pages landed in the audit's
  "no updated DATE" bucket and Quartz could not read the page date). 2026-08
  previous-month column corrected from $0.00 to $264.51 / 11 orders.

incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
  regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
  wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
  endpoint HTTP 401 (auth required, service up), no recurrence.

reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
  document no longer exists on disk, so the baseline cannot be re-verified against
  a live source. Flagged, not edited.

Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
2026-10-21 03:14:26 +08:00
Tony0410 fbd4f297eb wiki: publish pending edits (2026-10-20) 2026-10-20 18:38:11 +08:00
Tony0410 93c55efc94 stocktake 2026-10-19: skills-index 346/70; freshness 47 stale/43 fresh/7 no-date; open-claw VM 403 verified absent from PVE 2026-10-19 03:03:15 +08:00
Tony0410 dd53e5fbe1 stocktake 2026-10-18: skills-index 346 skills/70 cats; freshness 47 stale/41 fresh/7 no-date; quartz build skipped (no quartz.config.ts) 2026-10-18 03:01:17 +08:00
Tony0410 76de919e6a stocktake 2026-10-16: skills-index 346 skills/70 cats; freshness 47 stale/40 fresh/7 no-date; tailscale re-verified live — omarchymac (100.84.180.69) added, mattermost state corrected, offline ages refreshed (46 non-funnel nodes) 2026-10-16 03:04:25 +08:00
Tony0410 deb6a8eec2 stocktake 2026-10-14: SECOND Proxmox host m93p (100.104.51.98, PVE 9.2.0, unclustered, IDs collide) discovered — gitea/ntfy/freshrss/openweb were never on pve; tailnet inventory rebuilt 30 -> 45 nodes (15 missing); VM 282 running; Camofox recovered to 29.9M
skills-index 346 skills / 70 cats; freshness 48 stale / 38 fresh / 7 no-date of 93 pages

- infra/proxmox.md: two-host rewrite, m93p LXC+QEMU inventory, 4-entry correction table, VM 282 stopped->running, storage via pvesm (ssd0 88.17%/63GiB free)
- infra/tailscale.md: 45-node inventory rebuilt from live status; 23 funnel-ingress-node IPv6 peers noted+excluded; CT460 serve routes re-verified exactly (13 routes + :8443 Funnel + :9443)
- infra/network-map.md, current-state.md, index.md, log.md updated
- Open: 3 incidents (unverified pages left stale, not rewritten); omniroute/hosts/model-providers/docker-services still single-host in framing — m93p runs its own omniroute CT101, needs a dedicated pass
2026-10-14 03:13:04 +08:00
Tony0410 828913e23b audit(2026-10-14): 10-13 broke the clean streak — wiki-stocktake failed on a dead opencode credential (HTTP 404, incident 1a8e90_d11698bd7275); stranded skills-index + freshness-20261013 recovered; retention floor moved to 2026-10-08 07:11 2026-10-14 00:17:31 +08:00
hermes 0eee5f101e stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123 2026-10-12 03:17:10 +08:00
hermes d981e30128 stocktake(2026-10-11): skills-index 346 skills/70 cats; freshness 62 stale/24 fresh/7 no-date; 5 live-drift corrections
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
  CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
  Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
  registered as custom provider omni-101 (5384 models, default smart-route,
  carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
  CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
  Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
  undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
  (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
  (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
  VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
  nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
  no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
  hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
  CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.

Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.

62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.

Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
2026-10-11 03:11:07 +08:00
Hermes c11f11e652 stocktake 2026-10-10: skills-index + freshness report (62 stale/23 fresh/7 no-date of 92); current-state 10-10 section (timeout-storm window re-tallied 163/15, 10 open incidents, stocktake in-flight); tailscale.md verified vs live serve status; log entry
- Wiki maintenance run of wiki-maintain.sh, manual (scheduled 1a8e90ea2271 03:00 run in-flight)
- Verified live: tailscale serve status (all routes match), cron jobs.json (30 jobs), executions.db window, SKILL.md count
- Not verified: docker-services.md (docker.sock permission denied) — left stale; Quartz rebuild inert (no quartz.config.ts)
2026-10-10 03:05:36 +08:00
Tony0410 8cab69bbd0 wiki audit 2026-10-09: fleet-wide cron timeout storm (15 failures/10 jobs) + memory recall-relevance incident; backfill 2026-10-08 window; commit stranded stocktake artifacts 2026-10-10 00:27:15 +08:00
Tony0410 8c63ef24f0 chore: regenerate skills-index + freshness report 2026-10-07 2026-10-07 03:03:19 +08:00
Tony0410 33f4e95d95 wiki-maintain: regenerate skills-index and freshness stocktake 2026-10-06 2026-10-06 03:03:02 +08:00
Tony0410 b9f69d3f75 wiki audit 2026-10-03: cliprox second job confirmed (2/3), work-mailbox incident page, streak recount, skills-index 341->344 2026-10-03 00:18:51 +08:00
Tony0410 0c4ed3350b wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00
Tony0410 389de2119b Wiki maintenance 2026-09-25: regenerate skills-index (341 skills), freshness audit (67 stale pages) 2026-09-25 03:01:29 +08:00
Tony0410 2a082ca53d wiki-maintain.sh 2026-09-24: regenerate skills-index (341 skills), freshness audit (67 stale pages) 2026-09-24 03:03:00 +08:00
Tony0410 dd1c99c712 wiki-maintain: regenerate skills-index (341 skills, 73 categories) and add freshness report (67 stale pages) 2026-09-23 03:01:44 +08:00
Tony0410 775ed9fc78 Wiki maintenance 2026-09-22: regenerated skills-index (340 skills), freshness audit (67 stale pages) 2026-09-22 03:03:27 +08:00
Tony0410 345853108d Wiki maintenance 2026-09-21: skills-index regenerated, freshness audit, tailscale routes verified
- systems/skills-index.md: Regenerated from live skill tree (340 skills in 73 categories)
- raw/stocktake/freshness-20260921.txt: Freshness audit — 67 stale pages (≥21d), 5 fresh, 7 no-date
- infrastructure/tailscale.md: Verified against live tailscale serve status, reordered routes to match current state, updated verified_on to 2026-09-21
2026-09-21 03:02:29 +08:00
Tony0410 f9ac274bac Weekly maintenance: regenerate skills-index (340 skills), freshness audit 2026-09-20 (67 stale, 5 fresh, 7 no-date) 2026-09-20 03:02:05 +08:00
Tony0410 5e96254eab Wiki maintenance 2026-09-19: regenerate skills-index (340 skills/73 cats), update scheduled-tasks (26 jobs), add freshness report (68 stale pages) 2026-09-19 03:02:27 +08:00
Tony0410 0c046e1aa8 Wiki maintenance 2026-09-18: regenerated skills-index (340 skills/73 cats), freshness audit (66 stale pages) 2026-09-18 03:01:33 +08:00
Tony0410 8fea72e7c6 Wiki maintenance 2026-09-17: regenerated skills-index (340 skills/73 cats), freshness audit (66 stale pages) 2026-09-17 03:02:04 +08:00
Tony0410 e125396c92 wiki-maintain.sh: Regenerate skills-index.md (339 skills/73 cats) + freshness audit 2026-09-15 (66 stale pages) 2026-09-15 03:06:33 +08:00
Tony0410 3bfecb736d wiki-maintain.sh: regenerate skills-index (339 skills/73 cats), freshness audit 2026-09-14 (66 stale) 2026-09-14 03:01:50 +08:00
Tony0410 3cdda34422 wiki-maintain.sh: regenerate skills-index (339 skills/73 cats), add freshness reports 2026-09-12/13 2026-09-13 03:03:03 +08:00
Tony0410 17283295d3 Wiki maintenance 2026-09-11: regenerate skills-index (339 skills), add freshness report, create wiki-maintain.sh automation script 2026-09-11 03:05:21 +08:00
Tony0410 ae5724f657 Wiki maintenance 2026-09-10: regenerate skills-index, add freshness report, update tailscale.md routes 2026-09-10 03:04:02 +08:00
Tony0410 77250ea8c9 incident: SFTPGo outage 2026-09-02 — CT299 tailscaled death + stale lxc-attach recovery
- New incident page documenting CT299 SFTPGo outage (02:18 AWST)
- Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start
- Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299
- Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04
- Updated log.md with incident entry
- Updated index.md last-updated date

No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
2026-09-04 00:05:32 +08:00
Tony0410 f1ea8fd228 maintain: wiki stocktake refresh + scheduled-tasks cleanup 2026-08-16 03:08:09 +08:00
Tony0410 20a3164dca maintain: general wiki freshness mechanism + restore tailscale routes page
Replace the skills-only regeneration script with a general wiki-maintain.sh that:
- regenerates skills-index from live skill tree (330 skills / 26 categories)
- audits EVERY page for staleness vs its updated date (>21d flagged)
- rebuilds + redeploys the Quartz site
- writes a dated freshness report to raw/stocktake/

Also fix the tailscale.md page (it had drifted to only 4 routes; reality now has
11 CT460 routes incl /webui, /agent-estate, /synergy-prep, /evening-grid, /wiki
and the /wiki proxy was also missing from the atomic apply script — added there).

Cron wiki-stocktake-maintenance (1a8e90ea2271) now runs this daily 03:00 and
commits+pushes so the wiki cannot silently drift again.
2026-08-16 00:16:48 +08:00