diff --git a/current-state.md b/current-state.md index b61e045..ffe9f79 100644 --- a/current-state.md +++ b/current-state.md @@ -14,6 +14,32 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv # Current State Updated: 2026-10-21 +## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST) + +- **Scheduled wiki maintenance ran clean.** `wiki-maintain.sh` completed all three steps — skills-index + regenerated (346 → **347 skills**, new `australian-tax-return-review` under Finance), freshness audit written + (`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed, `hermes-wiki-serve.service` + restarted. Job `1a8e90ea2271` is `active`, `last_status: ok`, pinned `custom:omni-101` / `free-stack`. +- **Tailscale node inventory corrected — the table had drifted again.** `octop` (100.92.163.122) and `springroll` + (100.100.153.125) were **missing from the table entirely**; `openmausbot` / `openmausbot-vm` appeared in prose but + not in the table body; `omarchymac` had gone **offline** (was online); every offline age was stale. Rebuilt from + `tailscale status --json` **keyed on DNS name, not `HostName`** — the JSON `HostName` for the `freshrss` node is + `ct260`, and **both OmniRoute nodes report the hostname `omniroute`**, which collides silently. Now **51 non-funnel + nodes** (50 IPv4 peers + `hermes`). → [[infrastructure/tailscale]] +- **Two Uber Eats monthly reports were missing from the published wiki and have been recovered.** The generator wrote + only to `~/wiki/reports/uber-eats/` (the `~/wiki` repo), never to the canonical `agent-estate-wiki` tree — so + **2026-08 and 2026-09 never reached the published site**. **Root cause fixed:** the generator now targets the + canonical tree and mirrors to the legacy one, and now emits **YAML frontmatter** (without it the pages landed in the + audit's "no updated DATE" bucket and Quartz could not read the page date). Both months verified against + `~/.hermes/uber_eats_report.json`; the 2026-08 previous-month column corrected from $0.00 to **$264.51 / 11 orders**. + → [[reports/uber-eats]] +- **Two incident pages re-verified live and annotated:** [[incidents/2026-08-15-skills-index-drift]] (its named + `regenerate-skills-index.sh` no longer exists — regeneration now lives in `wiki-maintain.sh` step 1/3) and + [[incidents/2026-09-02-sftpgo-outage]] (**CT299 running**, health endpoint HTTP 401 = auth required, service up). +- **One page left stale on purpose:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source document + `/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists**, so the baseline + cannot be re-verified against a live source. Flagged, not edited. + ## Recent Changes (2026-10-21, nightly audit; live probes 2026-10-21 00:03-00:15 AWST) - **A clean day: 2026-10-20 closed 168 `completed` / 0 `failed`** (plus 1 `unknown`, plus this audit) — the first diff --git a/incidents/2026-08-15-skills-index-drift.md b/incidents/2026-08-15-skills-index-drift.md index 6ef21c5..9728096 100644 --- a/incidents/2026-08-15-skills-index-drift.md +++ b/incidents/2026-08-15-skills-index-drift.md @@ -3,8 +3,9 @@ title: "Skills Index Drift — 4x Undercount" type: incident status: resolved created: 2026-08-15 -updated: 2026-08-15 +updated: 2026-10-21 resolved_on: 2026-08-15 +verified_on: 2026-10-21 confidence: high tags: [skills, wiki, drift, auto-regeneration] sources: [session-investigation, agent-estate-wiki] @@ -25,7 +26,9 @@ Static documentation for a mutable inventory. The skills tree at `~/.hermes/skil ## Resolution 1. Regenerated `systems/skills-index.md` from the live skill tree (all 330 skills, 26 categories) -2. Created `regenerate-skills-index.sh` script for automated regeneration +2. Created a regeneration script for automated regeneration — **superseded on 2026-10-21:** the + regeneration now lives inside `wiki-maintain.sh` (step 1/3) in the wiki repo; no standalone + `regenerate-skills-index.sh` exists on disk any more. 3. Set up cron job `1a8e90ea2271` to run daily at 03:00 AWST 4. Script auto-commits and pushes to Gitea (`Anthony/agent-estate-wiki`) 5. Quartz site at `http://100.118.5.51:9120/systems/skills-index` rebuilt and redeployed @@ -33,6 +36,13 @@ Static documentation for a mutable inventory. The skills tree at `~/.hermes/skil ## Lesson Static wiki pages that reflect **mutable inventory** (skills, services, cron jobs) **MUST be auto-regenerated on a schedule**, not hand-maintained. The moment the estate grows, a hand-written page becomes stale. +## Re-verified 2026-10-21 +The prevention mechanism is live and working. `wiki-maintain.sh` step 1/3 regenerated +`systems/skills-index.md` from `~/.hermes/skills/` on the 2026-10-21 03:00 scheduled run: +**347 skills across 70 categories** (the "26 categories" figure in the original write-up was +the count at the time of the incident, not the current topology — the skill tree has since +nested deeper). The daily cron job `1a8e90ea2271` is `active` with `last_status: ok`. + ## Prevention - Daily cron job regenerates skills-index.md from live tree - Page includes "auto-regenerated, do not hand-edit" warning diff --git a/incidents/2026-09-02-sftpgo-outage.md b/incidents/2026-09-02-sftpgo-outage.md index c953ea6..a4253a8 100644 --- a/incidents/2026-09-02-sftpgo-outage.md +++ b/incidents/2026-09-02-sftpgo-outage.md @@ -3,8 +3,8 @@ title: SFTPGo Outage — CT299 Offline (2026-09-02) type: incident status: resolved created: 2026-09-04 -updated: 2026-09-04 -verified_on: 2026-09-04 +updated: 2026-10-21 +verified_on: 2026-10-21 confidence: high tags: [incident, sftpgo, ct299, tailscale, proxmox] sources: [session:20260902_014540_c193be] @@ -62,6 +62,12 @@ tailscale ping sftpgo curl -sS http://100.88.223.70:10080/healthz ``` +## Re-verified 2026-10-21 +Live state checked: **CT299 `sftpgo` is `running`** on `pve` (100.96.100.82) per `pct list`; +the node is `active` in `tailscale status` at `100.88.223.70`; the SFTPGo health endpoint +`http://100.88.223.70:10080/healthz` returns **HTTP 401** (expected — auth required, service up). +The 2026-09-02 outage remains resolved; no recurrence. + ## Lessons Learned 1. **Clean up `lxc-attach` probes** — always exit cleanly and verify no lingering processes after diagnostic sessions diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 11dba58..c6e5d3d 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-10-19 -verified_on: 2026-10-19 +updated: 2026-10-21 +verified_on: 2026-10-21 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,31 +19,32 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-10-19) -Full node list from `tailscale status` — **49 non-funnel nodes** (48 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. +## Tailscale Node Inventory (verified 2026-10-21) +Full node list from `tailscale status` — **51 non-funnel nodes** (50 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong). Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`. -Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. `tailscale status` reports 71 peers total, of which 23 are Tailscale's own entries (see the note below); the 48 with IPv4 addresses are the homelab set. +Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. +Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. | Node | IP | State | |---|---|---| | hermes | 100.118.5.51 | **self** | | agentzero | 100.106.54.94 | online | -| anthonys-imac | 100.112.123.124 | idle/offline 22d | +| anthonys-imac | 100.112.123.124 | idle/offline 27d | | anthonys-ipad | 100.111.198.120 | online | | anthonys-iphone | 100.100.116.67 | online | | anthonys-mac-mini | 100.74.17.15 | online | -| cachyos | 100.124.208.15 | idle/offline 42d | +| cachyos | 100.124.208.15 | idle/offline 47d | | clawtest | 100.122.189.81 | online | | cliproxyapi | 100.83.112.94 | online | -| cloudreve | 100.111.187.7 | idle/offline 83d | -| conduit | 100.81.79.87 | idle/offline 39d | -| docker-01 | 100.121.12.31 | online | +| cloudreve | 100.111.187.7 | idle/offline 88d | +| conduit | 100.81.79.87 | idle/offline 44d | | docker | 100.96.150.69 | online | -| elementsynapse | 100.75.250.37 | idle/offline 86d | +| docker-01 | 100.121.12.31 | online | +| elementsynapse | 100.75.250.37 | idle/offline 91d | | freellmapi | 100.127.208.75 | online | -| freshrss-new | 100.110.180.115 | idle/offline 59d | | freshrss | 100.83.60.93 | online | +| freshrss-new | 100.110.180.115 | idle/offline 64d | | gitea | 100.82.100.128 | online | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online | @@ -51,17 +52,20 @@ Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest- | litellm | 100.79.29.64 | online | | m93p | 100.104.51.98 | online | | mailflow | 100.115.60.92 | online | -| mattermost | 100.98.116.14 | active; relay "syd"; idle/offline 29d — CT255 stopped (re-verified live 2026-10-19) | +| mattermost | 100.98.116.14 | idle/offline 31d — CT255 stopped (re-verified live 2026-10-21) | | n8n | 100.84.124.30 | online | | nextcloud | 100.90.102.19 | online | | ntfy | 100.67.153.116 | active | +| octop | 100.92.163.122 | online | | ollama | 100.108.44.119 | online | | omarchy | 100.73.227.103 | online | -| omarchymac | 100.84.180.69 | online | | omarchybook | 100.107.9.100 | online | -| omni | 100.93.204.4 | idle/offline 59d | -| omniroute-1 | 100.78.63.20 | active | -| omniroute | 100.88.81.19 | online | +| omarchymac | 100.84.180.69 | idle/offline 3d | +| omni | 100.93.204.4 | idle/offline 64d — CT601 stopped (re-verified live 2026-10-21) | +| omniroute | 100.88.81.19 | online — CT600 (live OmniRoute; re-verified 2026-10-21) | +| omniroute-1 | 100.78.63.20 | active — CT101 on m93p (re-verified 2026-10-21) | +| openmausbot | 100.91.214.91 | online | +| openmausbot-vm | 100.77.107.30 | online | | openweb | 100.69.230.13 | online | | paseo | 100.72.188.93 | online | | pve | 100.96.100.82 | online | @@ -69,9 +73,10 @@ Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest- | rclone | 100.107.116.17 | online | | runtipi | 100.96.244.39 | online | | rustdesk | 100.95.227.57 | online | -| sftpgo | 100.88.223.70 | active | +| sftpgo | 100.88.223.70 | active — CT299 (re-verified live 2026-10-21) | | social-dashboard | 100.70.183.37 | online | -| ubuntu-server-xfce | 100.101.26.104 | idle/offline 55d | +| springroll | 100.100.153.125 | online | +| ubuntu-server-xfce | 100.101.26.104 | idle/offline 60d | | vaultwarden | 100.123.225.90 | online | | webserver | 100.82.40.23 | online | diff --git a/log.md b/log.md index 994fb17..49a2274 100644 --- a/log.md +++ b/log.md @@ -990,3 +990,36 @@ No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe st - **No secrets written.** The gateway token pasted in the 2026-10-20 desktop session was **not** transcribed into any page; both the incident and current-state entries describe the request without the credential. Host names, IPs, job/incident IDs, ports and probe status codes only. + +## [2026-10-21] stocktake | Scheduled wiki maintenance (cron `1a8e90ea2271`, 03:00 AWST) +- **Maintenance script ran clean** — `wiki-maintain.sh` completed all three steps: skills-index regenerated + (**346 → 347 skills**; new `australian-tax-return-review` under Finance), freshness audit written + (`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed to `public/`, + `hermes-wiki-serve.service` restarted. +- **[[infrastructure/tailscale]] — node table rebuilt from live `tailscale status --json`.** The 2026-10-19 + table had drifted again: **`octop` (100.92.163.122) and `springroll` (100.100.153.125) were missing entirely**, + `openmausbot` / `openmausbot-vm` were described in prose but absent from the table body, `omarchymac` had gone + offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on **DNS name** rather + than the JSON `HostName` — the latter is `ct260` for the `freshrss` node, and **both OmniRoute nodes share the + hostname `omniroute`**, which silently collides. Now **51 non-funnel nodes** (50 IPv4 peers + `hermes`). + Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): + CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. +- **[[reports/uber-eats]] — two missing monthly reports recovered.** The generator + (`~/.hermes/cron/scripts/generate-uber-eats-report.py`) wrote only to `~/wiki/reports/uber-eats/` (the `~/wiki` + repo), **never to the canonical `agent-estate-wiki` tree**, so the 2026-08 and 2026-09 reports were missing from + the published wiki. **Root cause fixed:** the script now writes to `~/workspace/agent-estate-wiki/reports/uber-eats/` + and mirrors to the legacy tree. It also **now emits YAML frontmatter** — without it the pages landed in the + "no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing + months copied in and verified against `~/.hermes/uber_eats_report.json` (2026-09: $51.29 / 2 orders; 2026-08: + $211.16 / 7 orders). The **2026-08 previous-month column was corrected** — the original run had no comparison + payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change. +- **[[incidents/2026-08-15-skills-index-drift]] and [[incidents/2026-09-02-sftpgo-outage]] re-verified against live + state** and annotated. The skills-drift page named a `regenerate-skills-index.sh` that **no longer exists on disk** + (regeneration now lives inside `wiki-maintain.sh` step 1/3); the SFTPGo page confirmed **CT299 running** and the + health endpoint returning **HTTP 401** (auth required, service up) — no recurrence. +- **Still stale, left alone deliberately:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source + document `/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists on disk**, + so the audit baseline cannot be re-verified against a live source. Flagged, not edited. +- **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`, + `model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00. +- **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes. diff --git a/raw/stocktake/freshness-20261021.txt b/raw/stocktake/freshness-20261021.txt new file mode 100644 index 0000000..c3e6c7f --- /dev/null +++ b/raw/stocktake/freshness-20261021.txt @@ -0,0 +1,114 @@ +# Wiki freshness audit — 2026-10-21 +Threshold: >21 days since `updated` + +## STALE (1) + +- [ 54d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28) + +## FRESH (94) + +- [ 1d] CONTRIBUTING.md +- [ 17d] agents/hermes-chaos.md +- [ 17d] agents/hermes-production.md +- [ 1d] agents/index.md +- [ 10d] agents/nanobot.md +- [ 1d] agents/open-claw.md +- [ 10d] agents/qwenpaw.md +- [ 1d] agents/zeroclaw.md +- [ 1d] comparisons/camoufox-vs-chrome.md +- [ 1d] comparisons/hermes-vs-nanobot.md +- [ 1d] comparisons/memory-backends.md +- [ 1d] comparisons/model-routers.md +- [ 8d] comparisons/steel-vs-camoufox.md +- [ 1d] concepts/agent-memory.md +- [ 8d] concepts/browser-automation.md +- [ 1d] concepts/model-routing.md +- [ 1d] concepts/persistent-shells.md +- [ 1d] concepts/tool-calling.md +- [ 0d] current-state.md +- [ 1d] decisions/index.md +- [ 1d] decisions/keep-openrouter-as-fallback.md +- [ 1d] decisions/persistent-shell-enabled.md +- [ 1d] decisions/rollback-steel-to-camofox.md +- [ 1d] experiments/index.md +- [ 1d] incidents/2026-08-01-mnemosyne-capture-failure.md +- [ 0d] incidents/2026-08-08-gitea-config-backup-push-dead.md +- [ 0d] incidents/2026-08-15-skills-index-drift.md +- [ 9d] incidents/2026-08-18-mnemosyne-capture-pipeline.md +- [ 18d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md +- [ 1d] incidents/2026-08-29-gmail-read-state.md +- [ 0d] incidents/2026-09-02-sftpgo-outage.md +- [ 10d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 15d] incidents/2026-10-01-cliproxy-connection-auth-expired.md +- [ 17d] incidents/2026-10-03-ct450-hermes-update.md +- [ 17d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 15d] incidents/2026-10-05-dead-model-pin-repin.md +- [ 8d] incidents/2026-10-09-cron-provider-timeout-storm.md +- [ 12d] incidents/2026-10-09-memory-recall-relevance.md +- [ 5d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md +- [ 0d] incidents/2026-10-16-omniroute-503-resource-pressure.md +- [ 2d] incidents/2026-10-18-degenerate-stream-canned-substitute.md +- [ 0d] incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md +- [ 1d] infrastructure/backups.md +- [ 1d] infrastructure/credentials-map.md +- [ 10d] infrastructure/docker-services.md +- [ 1d] infrastructure/domains-and-tunnels.md +- [ 17d] infrastructure/hosts.md +- [ 7d] infrastructure/network-map.md +- [ 7d] infrastructure/proxmox.md +- [ 0d] infrastructure/tailscale.md +- [ 9d] queries/retained-research-answers.md +- [ 0d] reports/uber-eats/2026-07.md +- [ 0d] reports/uber-eats/2026-08.md +- [ 0d] reports/uber-eats/2026-09.md +- [ 0d] reports/uber-eats/index.md +- [ 1d] runbooks/backup-wiki.md +- [ 1d] runbooks/diagnose-docker.md +- [ 1d] runbooks/gateway-resource-resilience.md +- [ 8d] runbooks/job-radar.md +- [ 1d] runbooks/multiplexer-setup.md +- [ 17d] runbooks/provider-health.md +- [ 1d] runbooks/recover-docker-service.md +- [ 1d] runbooks/restart-browser.md +- [ 1d] runbooks/restart-hermes.md +- [ 1d] runbooks/restore-openrouter.md +- [ 1d] runbooks/test-openrouter.md +- [ 1d] runbooks/troubleshoot-tool-loop.md +- [ 17d] runbooks/update-hermes-safely.md +- [ 8d] systems/browser-backend.md +- [ 1d] systems/cloudflare-tunnel.md +- [ 0d] systems/current-profile.md +- [ 1d] systems/guanaco.md +- [ 1d] systems/headless-server.md +- [ 10d] systems/hermes-agent.md +- [ 1d] systems/hermes-ui.md +- [ 9d] systems/home-assistant.md +- [ 1d] systems/homepage-dashboard.md +- [ 10d] systems/ignis.md +- [ 17d] systems/memory-backend.md +- [ 2d] systems/messaging-integrations.md +- [ 2d] systems/mnemosyne.md +- [ 20d] systems/model-providers.md +- [ 9d] systems/nanobot.md +- [ 1d] systems/nextcloud.md +- [ 1d] systems/obsidian.md +- [ 1d] systems/omniroute.md +- [ 1d] systems/opencode-go.md +- [ 1d] systems/openrouter.md +- [ 10d] systems/paseo.md +- [ 2d] systems/push-notifications.md +- [ 0d] systems/scheduled-tasks.md +- [ 0d] systems/skills-index.md +- [ 1d] systems/terminal-backend.md +- [ 1d] systems/tool-search.md + +## NO updated DATE (6) + +- SCHEMA.md +- experiments/active/example.md +- incidents/2026-07-29-mnemosyne-relay.md +- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md +- index.md +- log.md + +TOTAL: 1 stale, 94 fresh, 6 no-date of 101 pages \ No newline at end of file diff --git a/reports/uber-eats/2026-07.md b/reports/uber-eats/2026-07.md index f1efe09..2c53bcd 100644 --- a/reports/uber-eats/2026-07.md +++ b/reports/uber-eats/2026-07.md @@ -1,3 +1,14 @@ +--- +title: Uber Eats Monthly Spend — 2026-07 +type: report +status: active +created: 2026-08-01 +updated: 2026-10-21 +verified_on: 2026-10-21 +confidence: high +tags: [reports, uber-eats, finance] +--- + # 🍔 Uber Eats Monthly Spend — 2026-07 Generated: 2026-08-01T03:30:00+08:00 diff --git a/reports/uber-eats/2026-08.html b/reports/uber-eats/2026-08.html new file mode 100644 index 0000000..4886e09 --- /dev/null +++ b/reports/uber-eats/2026-08.html @@ -0,0 +1,85 @@ + + +
+ + +