stocktake 2026-10-12: skills-index 346 skills/70 cats; freshness 54 stale/32 fresh/7 no-date; nanobot + home-assistant re-verified live (nanobot web UI back up on :8766, OmniRoute :8900 still closed -> stays degraded; HA HTTP 200 on :8123

This commit is contained in:
hermes
2026-10-12 03:17:10 +08:00
parent 6fc109e546
commit 0eee5f101e
7 changed files with 161 additions and 14 deletions
+7
View File
@@ -14,6 +14,13 @@ sources: [live-config, live-cron-list, live-hermes-version]
# Current State # Current State
Updated: 2026-10-12 Updated: 2026-10-12
## Recent Changes (2026-10-12, 03:00 stocktake; live probes 2026-10-12 03:0x-03:1x AWST)
- **02 stocktake corrections from live probes.** [[systems/nanobot]] was DOWN as of 2026-08-07; its web UI on CT333 is **back up** (HTTP `200` on `100.122.189.81:8766`), though the OmniRoute port `8900` endpoint is still closed so `status` stays `degraded`. [[systems/home-assistant]] confirmed **up**: HTTP `200` on `100.68.139.5:8123`, VMID 100 running, Tailscale node offering an exit node — confidence raised low → medium. → [[systems/nanobot]], [[systems/home-assistant]]
- **Freshness snapshot: 54 stale / 32 fresh / 7 no-date of 93 pages** (threshold >21 days). The bulk of the stale list is the 2026-07-22 seeding cohort (concepts, runbooks, comparisons) and is structural, not a regression. → `raw/stocktake/freshness-20261012.txt`
- **Skills index regenerated: 346 skills / 70 categories**, unchanged count from the previous run. → [[systems/skills-index]]
- **Tailscale routes re-verified** against `tailscale serve status` — the CT460 table in [[infrastructure/tailscale]] matches live output exactly (13 tailnet routes + `:8443/webhook` Funnel + `:9443` Vite dev). Two nodes joined since 2026-10-11: `omarchy` (100.73.227.103) and `omniroute-1` (100.78.63.20, both active).
## Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST) ## Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST)
- **The cron fleet is clean for a second consecutive day: 2026-10-11 ran `171 completed / 0 failed`** (+1 `running` = this audit), after 10-07 `1` / 10-08 `2` / 10-09 `15` failed. 10-10 finalises at **`175 / 0`**. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` steady at **5**; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). The two streak-1 jobs (`dca8482e4f76`, `1b23c042e17d`) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → [[systems/scheduled-tasks]] - **The cron fleet is clean for a second consecutive day: 2026-10-11 ran `171 completed / 0 failed`** (+1 `running` = this audit), after 10-07 `1` / 10-08 `2` / 10-09 `15` failed. 10-10 finalises at **`175 / 0`**. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` steady at **5**; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). The two streak-1 jobs (`dca8482e4f76`, `1b23c042e17d`) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → [[systems/scheduled-tasks]]
+1 -1
View File
@@ -30,7 +30,7 @@
### Integrations ### Integrations
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]] - [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **5 carry a non-clean `last_status`** as of 2026-10-12 00:10. **Two consecutive clean days: 2026-10-11 ran `171 completed` / 0 `failed`** (2026-10-10 finalised `175 / 0`), ending the 2026-10-09 provider timeout storm (15 failed executions / 10 jobs; 7 of its incidents auto-closed on 2026-10-10, 2 of its jobs remain `alerted`). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. See [[incidents/2026-10-09-cron-provider-timeout-storm]]
- [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-07. Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10 and 10-11 stocktake runs both committed (`d981e30`)**) - [[systems/skills-index]] — Installed skills inventory (**346 skills**; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-12 by the stocktake run itself (`6fc109e` follow-on: 346 skills / 70 categories). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]; **regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed**)
## Incidents ## Incidents
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; 7 of its incidents auto-closed on 2026-10-10, **2 remain `alerted`** (`dca8482e4f76`, `1b23c042e17d` — both weekly jobs still carrying the Friday `RuntimeError: Request timed out.`)
+29
View File
@@ -505,3 +505,32 @@ and the skills-index blurb no longer claims no regeneration since 2026-10-07.
`current-state.md`, `index.md`, `log.md`. `current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only. No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections
**Script run.** `wiki-maintain.sh` completed all three stages: skills-index regenerated from the live
skill tree (**346 skills** across 70 categories, unchanged count), freshness audit written to
`raw/stocktake/freshness-20261012.txt` (**54 stale / 32 fresh / 7 no-date of 93 pages**), and the Quartz
step took its documented no-`quartz.config.ts` path (build skipped, `public/` unchanged — expected, not a
failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and
written its artifacts, so results were read from the report rather than assumed.
**Drift corrected from live probes (2 pages):**
1. **[[systems/nanobot]]** — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on
CT333 is **back up**, HTTP `200` on `http://100.122.189.81:8766/`. Port `8900` — the endpoint the
disabled OmniRoute `nanobot` provider connection uses — is **still closed**, so `status` stays
`degraded` rather than flipping to active. SSH to CT333 is unreachable from CT460, so
`whatsapp-bridge.service` was not re-verified.
2. **[[systems/home-assistant]]** — carried `confidence: low` plus a page-wide "cannot confirm HA runs"
caveat from 2026-07-22. Live: HTTP `200` on `http://100.68.139.5:8123/`, Tailscale node
`homeassistant` active and offering an exit node, VMID 100 running per `qm list`. Confidence raised
low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).
**Verified and left stale.** [[systems/openrouter]] and [[systems/nextcloud]] were probed shallowly and
nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.
**Pages touched:** `systems/nanobot.md`, `systems/home-assistant.md`, `systems/skills-index.md`,
`raw/stocktake/freshness-20261012.txt`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
+106
View File
@@ -0,0 +1,106 @@
# Wiki freshness audit — 2026-10-12
Threshold: >21 days since `updated`
## STALE (54)
- [ 82d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
- [ 82d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
- [ 82d] concepts/agent-memory.md (updated 2026-07-22)
- [ 82d] concepts/model-routing.md (updated 2026-07-22)
- [ 82d] concepts/persistent-shells.md (updated 2026-07-22)
- [ 82d] concepts/tool-calling.md (updated 2026-07-22)
- [ 82d] decisions/index.md (updated 2026-07-22)
- [ 82d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
- [ 82d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
- [ 82d] experiments/index.md (updated 2026-07-22)
- [ 82d] infrastructure/backups.md (updated 2026-07-22)
- [ 82d] infrastructure/credentials-map.md (updated 2026-07-22)
- [ 82d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
- [ 82d] infrastructure/network-map.md (updated 2026-07-22)
- [ 82d] runbooks/backup-wiki.md (updated 2026-07-22)
- [ 82d] runbooks/diagnose-docker.md (updated 2026-07-22)
- [ 82d] runbooks/recover-docker-service.md (updated 2026-07-22)
- [ 82d] runbooks/restart-browser.md (updated 2026-07-22)
- [ 82d] runbooks/restart-hermes.md (updated 2026-07-22)
- [ 82d] runbooks/restore-openrouter.md (updated 2026-07-22)
- [ 82d] runbooks/test-openrouter.md (updated 2026-07-22)
- [ 82d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
- [ 82d] systems/cloudflare-tunnel.md (updated 2026-07-22)
- [ 82d] systems/headless-server.md (updated 2026-07-22)
- [ 82d] systems/home-assistant.md (updated 2026-07-22)
- [ 82d] systems/openrouter.md (updated 2026-07-22)
- [ 82d] systems/terminal-backend.md (updated 2026-07-22)
- [ 79d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
- [ 79d] systems/tool-search.md (updated 2026-07-25)
- [ 78d] systems/homepage-dashboard.md (updated 2026-07-26)
- [ 78d] systems/obsidian.md (updated 2026-07-26)
- [ 76d] runbooks/job-radar.md (updated 2026-07-28)
- [ 76d] runbooks/multiplexer-setup.md (updated 2026-07-28)
- [ 75d] systems/opencode-go.md (updated 2026-07-29)
- [ 74d] systems/hermes-ui.md (updated 2026-07-30)
- [ 72d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
- [ 71d] comparisons/steel-vs-camoufox.md (updated 2026-08-02)
- [ 71d] concepts/browser-automation.md (updated 2026-08-02)
- [ 71d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
- [ 71d] reports/uber-eats/index.md (updated 2026-08-02)
- [ 71d] systems/browser-backend.md (updated 2026-08-02)
- [ 66d] systems/nanobot.md (updated 2026-08-07)
- [ 61d] comparisons/memory-backends.md (updated 2026-08-12)
- [ 61d] comparisons/model-routers.md (updated 2026-08-12)
- [ 59d] systems/guanaco.md (updated 2026-08-14)
- [ 58d] agents/index.md (updated 2026-08-15)
- [ 58d] agents/open-claw.md (updated 2026-08-15)
- [ 58d] agents/zeroclaw.md (updated 2026-08-15)
- [ 58d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
- [ 57d] CONTRIBUTING.md (updated 2026-08-16)
- [ 56d] systems/nextcloud.md (updated 2026-08-17)
- [ 45d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
- [ 44d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
- [ 38d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
## FRESH (32)
- [ 8d] agents/hermes-chaos.md
- [ 8d] agents/hermes-production.md
- [ 1d] agents/nanobot.md
- [ 1d] agents/qwenpaw.md
- [ 0d] current-state.md
- [ 0d] incidents/2026-08-18-mnemosyne-capture-pipeline.md
- [ 9d] incidents/2026-08-26-work-outlook-basic-auth-disabled.md
- [ 1d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 6d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 8d] incidents/2026-10-03-ct450-hermes-update.md
- [ 8d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 6d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 3d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 3d] incidents/2026-10-09-memory-recall-relevance.md
- [ 1d] infrastructure/docker-services.md
- [ 8d] infrastructure/hosts.md
- [ 0d] infrastructure/proxmox.md
- [ 1d] infrastructure/tailscale.md
- [ 0d] queries/retained-research-answers.md
- [ 8d] runbooks/provider-health.md
- [ 8d] runbooks/update-hermes-safely.md
- [ 10d] systems/current-profile.md
- [ 1d] systems/hermes-agent.md
- [ 1d] systems/ignis.md
- [ 8d] systems/memory-backend.md
- [ 1d] systems/messaging-integrations.md
- [ 0d] systems/mnemosyne.md
- [ 11d] systems/model-providers.md
- [ 1d] systems/omniroute.md
- [ 1d] systems/paseo.md
- [ 0d] systems/scheduled-tasks.md
- [ 0d] systems/skills-index.md
## NO updated DATE (7)
- SCHEMA.md
- experiments/active/example.md
- incidents/2026-07-29-mnemosyne-relay.md
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
- index.md
- log.md
- reports/uber-eats/2026-07.md
TOTAL: 54 stale, 32 fresh, 7 no-date of 93 pages
+10 -6
View File
@@ -3,9 +3,9 @@ title: Home Assistant
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-07-22 updated: 2026-10-12
verified_on: 2026-07-22 verified_on: 2026-10-12
confidence: low confidence: medium
tags: [system, automation] tags: [system, automation]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
--- ---
@@ -16,12 +16,13 @@ sources: [raw/configs/hermes-config-sanitized.txt]
Home automation and device control; one of Hermes's messaging/integration platforms. Home automation and device control; one of Hermes's messaging/integration platforms.
## Location ## Location
- **Verified (homelab infra docs, not this host):** VM 100 on `pve`, hostname `haos14.0` (Home Assistant OS), internal IP `172.30.232.1`. - **Verified 2026-10-12 (live HTTP):** Home Assistant OS answers HTTP `200` on `http://100.68.139.5:8123/` over Tailscale, and the Tailscale node `homeassistant` (`100.68.139.5`) is **active** offering an exit node. The service is up and reachable on the tailnet.
- **Verified (homelab infra docs, not this host):** VM 100 on `pve`, hostname `haos14.0` (Home Assistant OS), internal IP `172.30.232.1`. `qm list` confirms VMID 100 `homeassistant` is **running** (13 GB RAM).
- **Verified (Hermes config):** `homeassistant` and `hermes-homeassistant` appear in Hermes's platform/display integration lists (`~/.hermes/config.yaml`), alongside `qqbot`, `signal`, `slack`, `teams`, `telegram`. - **Verified (Hermes config):** `homeassistant` and `hermes-homeassistant` appear in Hermes's platform/display integration lists (`~/.hermes/config.yaml`), alongside `qqbot`, `signal`, `slack`, `teams`, `telegram`.
- **Unknown:** the actual connection details (HA base URL, long-lived access token, which entities/automations Hermes can reach) are not present in the sanitized config snapshot — likely held in a plugin-specific credentials file not captured there, or not yet wired up beyond being enabled as a platform. - **Unknown:** the actual connection details (HA base URL, long-lived access token, which entities/automations Hermes can reach) are not present in the sanitized config snapshot — likely held in a plugin-specific credentials file not captured there, or not yet wired up beyond being enabled as a platform.
## Current configuration ## Current configuration
Not verified beyond the platform being listed as available/connected in `current-state.md`'s "WebUI connected platforms" line. No automations, entity lists, or voice-assistant config confirmed from this host. **2026-10-12:** HA itself is confirmed up (HTTP 200 on `:8123`, VM running). Still **not** verified: that Hermes can actually reach an entity — no automation, entity list, or voice-assistant config has been confirmed from this host, and no HA credentials were found on CT460.
## Connected systems ## Connected systems
- [[systems/hermes-agent]] - [[systems/hermes-agent]]
@@ -30,7 +31,10 @@ Not verified beyond the platform being listed as available/connected in `current
- [[systems/homepage-dashboard]] — Shown in dashboard More tab - [[systems/homepage-dashboard]] — Shown in dashboard More tab
## Known issues ## Known issues
- Genuine gap, not a "known issue" per se: this page cannot confirm HA is *actively* used by Hermes vs. merely enabled — worth verifying directly (e.g. a live call to a known HA entity) next time someone is in this area. - **Partially closed 2026-10-12:** HA itself is confirmed running and reachable (HTTP 200 on `100.68.139.5:8123`). Still open: whether Hermes can actually *use* it — a live call to a known HA entity would settle it, and requires the HA credentials, which are not on CT460.
## Change history
- 2026-10-12: HA service confirmed UP — HTTP 200 on tailnet `:8123`, Tailscale node `homeassistant` active/exit-node, VMID 100 running per `qm list`. Confidence raised low → medium. Hermes↔HA entity access remains unverified.
## Relevant runbooks ## Relevant runbooks
None yet — would be a good candidate once the connection is verified (e.g. `runbooks/restart-home-assistant-bridge`). None yet — would be a good candidate once the connection is verified (e.g. `runbooks/restart-home-assistant-bridge`).
+6 -5
View File
@@ -3,8 +3,8 @@ title: Nanobot
type: system type: system
status: degraded status: degraded
created: 2026-07-22 created: 2026-07-22
updated: 2026-08-07 updated: 2026-10-12
verified_on: 2026-08-07 verified_on: 2026-10-12
confidence: high confidence: high
tags: [system, nanobot, network] tags: [system, nanobot, network]
sources: [live-probe, omniroute-investigation] sources: [live-probe, omniroute-investigation]
@@ -16,9 +16,9 @@ Role in Anthony's setup.
## Location ## Location
- **Verified 2026-07-22:** LXC 333, hostname `clawtest`, Tailscale IP `100.122.189.81`. - **Verified 2026-07-22:** LXC 333, hostname `clawtest`, Tailscale IP `100.122.189.81`.
## Current state ## Current state
- **Status: DOWN (as of 2026-08-07).** Connection refused on port 8900 — the OmniRoute provider connection to nanobot (`58debd59`, prefix `nanobot`) was causing event-loop stalls because every model sync + credential health check retried it. Connection disabled in OmniRoute. Nanobot on CT333 is not responding. - **Status: DEGRADED (re-verified 2026-10-12).** The Nanobot web UI on CT333 is **UP again**: HTTP `200` on `http://100.122.189.81:8766/` serving the nanobot web UI (verified this audit). However port `8900` — the endpoint the OmniRoute `nanobot` provider connection (`58debd59`, prefix `nanobot`) uses — is still **closed**, so the OmniRoute integration remains disabled. Contrast with 2026-08-07, when connection was refused on 8900 and no service was confirmed listening anywhere.
- **Prior state:** `nanobot` process listening on port `8766` (verified via `ss -tlnp` on ct333, 2026-07-22). - **Prior state:** `nanobot` process listening on port `8766` (verified via `ss -tlnp` on ct333, 2026-07-22); DOWN 2026-08-07 (connection refused on 8900 — OmniRoute event-loop stalls from model sync + credential health-check retries; provider connection disabled in OmniRoute).
- `whatsapp-bridge.service` (systemd, active) — "WhatsApp Bridge for Nanobot" — also runs on ct333. Status unknown as of 2026-08-07. - `whatsapp-bridge.service` (systemd, active) — "WhatsApp Bridge for Nanobot" — also runs on ct333. Status not re-verified on 2026-10-12 (SSH to CT333 not reachable from CT460: connection timed out on 22/tcp; verification above is TCP/HTTP probing only).
- Publicly reachable via the same Cloudflare Tunnel as Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, `nano-ssh.martinwa.org` → SSH on the same host. See [[infrastructure/domains-and-tunnels]]. - Publicly reachable via the same Cloudflare Tunnel as Hermes: `nanobot.martinwa.org` → `100.122.189.81:8766`, `nano-ssh.martinwa.org` → SSH on the same host. See [[infrastructure/domains-and-tunnels]].
## Current state as of 2026-07-22 (superseded — see below) ## Current state as of 2026-07-22 (superseded — see below)
@@ -45,6 +45,7 @@ Role in Anthony's setup.
## Change history ## Change history
- 2026-07-22: audit corrected "not running" claim after direct verification on ct333. - 2026-07-22: audit corrected "not running" claim after direct verification on ct333.
- 2026-08-07: status changed to DEGRADED — nanobot confirmed DOWN (connection refused on 8900). OmniRoute provider connection disabled. CT333 service status unknown. - 2026-08-07: status changed to DEGRADED — nanobot confirmed DOWN (connection refused on 8900). OmniRoute provider connection disabled. CT333 service status unknown.
- 2026-10-12: web UI confirmed back up (HTTP 200 on `100.122.189.81:8766`, tailnet IP `100.122.189.81` = tailscale node `clawtest`). Status stays DEGRADED because the OmniRoute port 8900 endpoint is still closed. `whatsapp-bridge.service` not re-verified (SSH to CT333 unreachable from CT460).
## Relevant runbooks ## Relevant runbooks
- [[runbooks/restart-hermes]] - [[runbooks/restart-hermes]]
+2 -2
View File
@@ -3,8 +3,8 @@ title: Installed Skills
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-11 updated: 2026-10-12
verified_on: 2026-10-11 verified_on: 2026-10-12
confidence: high confidence: high
tags: [skills, hermes] tags: [skills, hermes]
sources: [] sources: []