stocktake 2026-11-20: re-verify 9-page 2026-10-29 stale cohort live; correct nanobot docker/cron drift, camofox host/port claims, dead-pin residual; freshness 0 stale/136 fresh

This commit is contained in:
Tony0410
2026-11-20 03:11:58 +08:00
parent 8bf0a37022
commit f603c29d17
11 changed files with 137 additions and 44 deletions
+40 -15
View File
@@ -3,8 +3,8 @@ title: Nanobot
type: agent type: agent
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [agent, nanobot, homelab] tags: [agent, nanobot, homelab]
sources: [live-probe, runtime-config] sources: [live-probe, runtime-config]
@@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
- **Workspace:** `/home/nano/.nanobot/workspace` - **Workspace:** `/home/nano/.nanobot/workspace`
## Current state ## Current state
- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening. - **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.)
- **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765` - **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765`
- **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766` - **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766`
- **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed. - **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed.
@@ -38,22 +38,31 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat
Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)* Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)*
## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29) ## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29)
`gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`. ~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago).
**Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]] **Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]]
## Cron jobs ## Cron jobs
Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded: **Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled**
- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST (all schedules Australia/Perth):
- `morning-quota-report` — daily at 07:30 AWST
- `daily-quota-730am` — daily at 07:30 AWST | Job | Schedule | Enabled |
- `vault-daily-seed` — daily at 08:00 AWST |---|---|---|
- `daily-agent-briefing` — daily at 08:15 AWST | `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ |
- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST | `nightly-skills-export-sync` | daily 04:00 | ✅ |
- `daily-quota-7pm` — daily at 19:00 AWST | `morning-news-podcast` | daily 07:00 | ✅ |
- `evening-quota-report` — daily at 21:00 AWST | `daily-quota-730am` | daily 07:30 | ✅ |
- `heartbeat` — every 30m | `morning-quota-report` | daily 07:30 | ❌ disabled |
- `dream` — every 2h (memory consolidation) | `vault-daily-seed` | daily 08:00 | ✅ |
| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) |
| `daily-quota-7pm` | daily 19:00 | ✅ |
| `evening-quota-report` | daily 21:00 | ❌ disabled |
| `heartbeat` | every 30m | ✅ |
| `dream` | every 2h (memory consolidation) | ✅ |
~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at
some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` /
`daily-quota-7pm` remains enabled, per Anthony's standing preference).
## Skills ## Skills
Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`. Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`.
@@ -73,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli
- [[runbooks/restart-hermes]] - [[runbooks/restart-hermes]]
- [[runbooks/nanobot-restart]] - [[runbooks/nanobot-restart]]
> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy:
> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`;
> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still
> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api,
> agentmail-daemon and readlater (:3210) processes all still run.
>
> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page
> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is
> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not
> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`,
> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant`
> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000.
>
> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the
> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`):
> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above.
## Related ## Related
- [[comparisons/hermes-vs-nanobot]] - [[comparisons/hermes-vs-nanobot]]
+12 -2
View File
@@ -3,8 +3,8 @@ title: Camoufox vs Chrome
type: comparison type: comparison
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: medium confidence: medium
tags: [comparison, browser] tags: [comparison, browser]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -46,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool.
- Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing. - Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.
- `cua-driver` bridges Hermes's computer-use tool calls to the browser. - `cua-driver` bridges Hermes's computer-use tool calls to the browser.
> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`,
> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`,
> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`.
> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin`
> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox).
> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ —
> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the
> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s
> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP).
> No Chrome/Chromium process found in this probe either.
### Chrome (not found in use) ### Chrome (not found in use)
- No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit. - No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit.
- Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome. - Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome.
+9 -2
View File
@@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox
type: comparison type: comparison
status: active status: active
created: 2026-07-29 created: 2026-07-29
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [comparison, browser, steel, camofox, decision] tags: [comparison, browser, steel, camofox, decision]
sources: [live-probe, session-history] sources: [live-probe, session-history]
@@ -77,6 +77,13 @@ the CT201 and CT333 deployments torn down — so returning to Steel means redepl
- **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus - **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus
`/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io.
> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel
> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084
> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`,
> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000.
> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok`
> (idle — `browserConnected:false` at probe time, no active session), unit active, config
> `cloud_provider: camofox`. All removal claims **hold**.
## Related ## Related
- [[systems/browser-backend]] - [[systems/browser-backend]]
- [[concepts/browser-automation]] - [[concepts/browser-automation]]
+6 -2
View File
@@ -3,8 +3,8 @@ title: Browser Automation
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [concept, browser, camofox, steel] tags: [concept, browser, camofox, steel]
sources: [live-probe, session-history] sources: [live-probe, session-history]
@@ -33,6 +33,10 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere
**Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]]. **Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`),
> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080.
> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere
> (CT201/CT333 containers, images, routes, plugin all re-probed absent).
## Related ## Related
- [[systems/browser-backend]] - [[systems/browser-backend]]
- [[comparisons/steel-vs-camoufox]] - [[comparisons/steel-vs-camoufox]]
+6 -2
View File
@@ -3,8 +3,8 @@ title: Decisions
type: collection type: collection
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [decisions] tags: [decisions]
--- ---
@@ -31,6 +31,10 @@ Stable strategic and config decisions, with evidence and status.
> plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole > plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole
> backend. > backend.
> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live
> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11
> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s
> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide.
## History ## History
- **Verified:** Decision pages initialized during wiki setup on 2026-07-22 - **Verified:** Decision pages initialized during wiki setup on 2026-07-22
- **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing - **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing
+11 -3
View File
@@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox
type: decision type: decision
status: active status: active
created: 2026-08-02 created: 2026-08-02
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [decision, browser, steel, camofox, rollback] tags: [decision, browser, steel, camofox, rollback]
sources: [session-history, live-config] sources: [session-history, live-config]
@@ -50,7 +50,7 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat
> restoration was needed or wanted. > restoration was needed or wanted.
## Rationale ## Rationale
1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. 1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.
2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports. 2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.
3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem. 3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.
4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain. 4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.
@@ -103,6 +103,14 @@ removed**, so the `plugin_steel` toolset is no longer selectable.
The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it" The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it"
assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]]. assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]].
> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed:
> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel
> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443,
> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact
> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's
> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service
> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13.
> The decision and its consequences are unchanged.
## Related ## Related
- [[comparisons/steel-vs-camoufox]] - [[comparisons/steel-vs-camoufox]]
- [[systems/browser-backend]] - [[systems/browser-backend]]
@@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
type: incident type: incident
status: resolved status: resolved
created: 2026-10-02 created: 2026-10-02
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [cron, omniroute, provider, auth, incident, resolved] tags: [cron, omniroute, provider, auth, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050]
@@ -130,6 +130,12 @@ done.
- **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model - **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model
name is absent from the provider's live catalog. name is absent from the provider's live catalog.
> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live:
> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`);
> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` —
> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three
> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on
> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands.
## Related ## Related
- [[incidents/2026-10-05-dead-model-pin-repin]] - [[incidents/2026-10-05-dead-model-pin-repin]]
+13 -2
View File
@@ -3,8 +3,8 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route
type: incident type: incident
status: resolved status: resolved
created: 2026-10-06 created: 2026-10-06
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
confidence: high confidence: high
tags: [cron, model-pin, omniroute, incident, resolved] tags: [cron, model-pin, omniroute, incident, resolved]
sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c]
@@ -119,6 +119,17 @@ state rather than left on trust.
- The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can - The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can
run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day. run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day.
> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned
> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads
> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual
> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200
> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both →
> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged).
> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`**
> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403
> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now
> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in
> [[systems/scheduled-tasks]], not here.
## Related ## Related
- [[systems/scheduled-tasks]] - [[systems/scheduled-tasks]]
+13
View File
@@ -1877,3 +1877,16 @@ recorded here so the next audit does not re-litigate it.
- **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**. - **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**.
- **Freshness at publish time** (`raw/stocktake/freshness-20261120.txt`): **9 stale / 127 fresh / 3 no-date of 139 pages**. All nine carry the *same* `updated: 2026-10-29` — one 22-day bulk cohort, not nine findings: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. **Not re-verified by this audit** — the live-vs-page re-verification sweep belongs to the 03:00 stocktake (pattern 23), so the cohort is recorded here rather than silently inherited. Five further pages sit at exactly 21d and become the 11-21 stale list: `incidents/2026-10-27-job-radar-budget-exhausted.md`, `incidents/2026-10-28-job-radar-stage1-iteration-cap.md`, `incidents/2026-10-29-conduit-voice-silence.md`, `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md`, `systems/homepage-dashboard.md`. - **Freshness at publish time** (`raw/stocktake/freshness-20261120.txt`): **9 stale / 127 fresh / 3 no-date of 139 pages**. All nine carry the *same* `updated: 2026-10-29` — one 22-day bulk cohort, not nine findings: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. **Not re-verified by this audit** — the live-vs-page re-verification sweep belongs to the 03:00 stocktake (pattern 23), so the cohort is recorded here rather than silently inherited. Five further pages sit at exactly 21d and become the 11-21 stale list: `incidents/2026-10-27-job-radar-budget-exhausted.md`, `incidents/2026-10-28-job-radar-stage1-iteration-cap.md`, `incidents/2026-10-29-conduit-voice-silence.md`, `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md`, `systems/homepage-dashboard.md`.
- **Published (patterns 13/30):** `wiki-maintain.sh` regenerated `systems/skills-index.md` (**299 skills / 67 categories**) and rebuilt Quartz — 139 pages parsed, 867 files emitted to `public/`, `hermes-wiki-serve.service` restarted (`systemctl --user is-active` → `active`, listener on `:9120`). Live check after the rebuild: `index`, `current-state`, `systems/mnemosyne`, `systems/scheduled-tasks`, `systems/skills-index` and all three incident pages return **200** on `http://127.0.0.1:9120/<path>`. Both vault conventions synced from the authoring tree with per-file `sha256` matches (8 files × 2 conventions; freshness report included), committed locally — the vault repo has no remote. - **Published (patterns 13/30):** `wiki-maintain.sh` regenerated `systems/skills-index.md` (**299 skills / 67 categories**) and rebuilt Quartz — 139 pages parsed, 867 files emitted to `public/`, `hermes-wiki-serve.service` restarted (`systemctl --user is-active` → `active`, listener on `:9120`). Live check after the rebuild: `index`, `current-state`, `systems/mnemosyne`, `systems/scheduled-tasks`, `systems/skills-index` and all three incident pages return **200** on `http://127.0.0.1:9120/<path>`. Both vault conventions synced from the authoring tree with per-file `sha256` matches (8 files × 2 conventions; freshness report included), committed locally — the vault repo has no remote.
## 2026-11-20 03:0x — Stocktake (cron 1a8e90ea2271): 9-page stale cohort re-verified live
- Ran `wiki-maintain.sh` (repo script): skills-index regenerated (**299 skills / 67 categories**), freshness report `raw/stocktake/freshness-20261120.txt` → **9 stale / 127 fresh / 3 no-date of 139**. All nine stale pages were the single 2026-10-29 cohort (22d) flagged by the 01:13 audit — treated as one verification sweep (pattern 23), not nine investigations.
- **Live probes:** `~/.hermes/config.yaml` browser block; `ss -ltnp` + process list + `cua-driver --version` on CT460; `camofox-browser.service` (`systemctl --user` → active) + `:9377/health` → ok; CT333 via PVE (`ss`, `ps`, `docker ps -a`, `tailscale ip -4`, gateway `/health`, `workspace/cron/jobs.json`); CT201 via PVE (`docker ps -a`, images, Serve status, `_archived-steel/`, `:8085` probe); OmniRoute chat-completions on both nodes (`cliprox/terra`, `free-stack`, `smart-route`); `command-code/gpt-5.6-terra`; `jobs.json` fleet read.
- **All 9 pages re-verified** (`updated`/`verified_on` → 2026-11-20) with dated notes: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`.
- **Corrections (claims disproved or drifted):**
- `agents/nanobot.md`: Docker list drifted — `tender_brattain` → `ecstatic_swartz` (random-name rename class), `quirky_darwin` removed (absent with `-a`), stopped `qdrant` present. Gateway PID now 1256104 (v0.4.1, `/health` ok). **Cron section rewritten from live scheduler**: 11 jobs / 8 enabled; `morning-quota-report`, `daily-agent-briefing`, `evening-quota-report` disabled; `rss-check-proxmox-selfhst` gone; new `nightly-skills-export-sync` (04:00) and `morning-news-podcast` (07:00). Quota pair 07:30/19:00 remains enabled. Scheduler file is `workspace/cron/jobs.json` (the `~/.nanobot/cron/` dir is empty).
- `comparisons/camoufox-vs-chrome.md`: struck the stale "9377 not listening on CT460 / service on CT450" claim — `:9377` **is** listening on CT460 (local `camofox-browser.service`, active); ws-server also on `:9222`.
- `decisions/rollback-steel-to-camofox.md`: Rationale's "(CT450)" host attribution corrected to **CT460** with strikethrough.
- `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27).
- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists).
- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced.
+12 -12
View File
@@ -1,37 +1,34 @@
# Wiki freshness audit — 2026-11-20 # Wiki freshness audit — 2026-11-20
Threshold: >21 days since `updated` Threshold: >21 days since `updated`
## STALE (9) ## STALE (0)
- [ 22d] agents/nanobot.md (updated 2026-10-29)
- [ 22d] comparisons/camoufox-vs-chrome.md (updated 2026-10-29)
- [ 22d] comparisons/steel-vs-camoufox.md (updated 2026-10-29)
- [ 22d] concepts/browser-automation.md (updated 2026-10-29)
- [ 22d] decisions/index.md (updated 2026-10-29)
- [ 22d] decisions/rollback-steel-to-camofox.md (updated 2026-10-29)
- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-29)
- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-29)
- [ 22d] runbooks/recover-docker-service.md (updated 2026-10-29)
## FRESH (127) ## FRESH (136)
- [ 9d] CONTRIBUTING.md - [ 9d] CONTRIBUTING.md
- [ 12d] agents/hermes-chaos.md - [ 12d] agents/hermes-chaos.md
- [ 12d] agents/hermes-production.md - [ 12d] agents/hermes-production.md
- [ 9d] agents/index.md - [ 9d] agents/index.md
- [ 0d] agents/nanobot.md
- [ 9d] agents/open-claw.md - [ 9d] agents/open-claw.md
- [ 16d] agents/qwenpaw.md - [ 16d] agents/qwenpaw.md
- [ 9d] agents/zeroclaw.md - [ 9d] agents/zeroclaw.md
- [ 0d] comparisons/camoufox-vs-chrome.md
- [ 9d] comparisons/hermes-vs-nanobot.md - [ 9d] comparisons/hermes-vs-nanobot.md
- [ 9d] comparisons/memory-backends.md - [ 9d] comparisons/memory-backends.md
- [ 9d] comparisons/model-routers.md - [ 9d] comparisons/model-routers.md
- [ 0d] comparisons/steel-vs-camoufox.md
- [ 9d] concepts/agent-memory.md - [ 9d] concepts/agent-memory.md
- [ 0d] concepts/browser-automation.md
- [ 9d] concepts/model-routing.md - [ 9d] concepts/model-routing.md
- [ 9d] concepts/persistent-shells.md - [ 9d] concepts/persistent-shells.md
- [ 9d] concepts/tool-calling.md - [ 9d] concepts/tool-calling.md
- [ 0d] current-state.md - [ 0d] current-state.md
- [ 0d] decisions/index.md
- [ 9d] decisions/keep-openrouter-as-fallback.md - [ 9d] decisions/keep-openrouter-as-fallback.md
- [ 9d] decisions/persistent-shell-enabled.md - [ 9d] decisions/persistent-shell-enabled.md
- [ 0d] decisions/rollback-steel-to-camofox.md
- [ 9d] experiments/index.md - [ 9d] experiments/index.md
- [ 8d] incidents/2026-07-29-mnemosyne-relay.md - [ 8d] incidents/2026-07-29-mnemosyne-relay.md
- [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md - [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md
@@ -43,8 +40,10 @@ Threshold: >21 days since `updated`
- [ 9d] incidents/2026-08-29-gmail-read-state.md - [ 9d] incidents/2026-08-29-gmail-read-state.md
- [ 8d] incidents/2026-09-02-sftpgo-outage.md - [ 8d] incidents/2026-09-02-sftpgo-outage.md
- [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md - [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md
- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
- [ 2d] incidents/2026-10-03-ct450-hermes-update.md - [ 2d] incidents/2026-10-03-ct450-hermes-update.md
- [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md - [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md
- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md
- [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md - [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md
- [ 19d] incidents/2026-10-09-memory-recall-relevance.md - [ 19d] incidents/2026-10-09-memory-recall-relevance.md
- [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md - [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md
@@ -107,6 +106,7 @@ Threshold: >21 days since `updated`
- [ 13d] runbooks/job-radar.md - [ 13d] runbooks/job-radar.md
- [ 9d] runbooks/multiplexer-setup.md - [ 9d] runbooks/multiplexer-setup.md
- [ 2d] runbooks/provider-health.md - [ 2d] runbooks/provider-health.md
- [ 0d] runbooks/recover-docker-service.md
- [ 9d] runbooks/restart-browser.md - [ 9d] runbooks/restart-browser.md
- [ 9d] runbooks/restart-hermes.md - [ 9d] runbooks/restart-hermes.md
- [ 9d] runbooks/restore-openrouter.md - [ 9d] runbooks/restore-openrouter.md
@@ -149,4 +149,4 @@ Threshold: >21 days since `updated`
- experiments/active/example.md - experiments/active/example.md
- log.md - log.md
TOTAL: 9 stale, 127 fresh, 3 no-date of 139 pages TOTAL: 0 stale, 136 fresh, 3 no-date of 139 pages
+7 -2
View File
@@ -3,8 +3,8 @@ title: Recover a Docker service
type: runbook type: runbook
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-10-29 updated: 2026-11-20
verified_on: 2026-10-29 verified_on: 2026-11-20
tags: [runbook, docker, recovery] tags: [runbook, docker, recovery]
sources: [] sources: []
confidence: high confidence: high
@@ -20,6 +20,11 @@ Container shows healthy after recovery probe reboots.
## Last successfully used ## Last successfully used
2026-07-22 2026-07-22
> **Re-verified 2026-11-20 (stocktake, procedure only).** `docker ps` as `hermes` on CT460 still
> returns permission denied (`/var/run/docker.sock` is `root:docker 0660`) — sudo or SSH to the
> owning host remains required. CT201 homepage degradation **persists**: `:8085` answers **502**
> locally, `homepage-manager` up but no dashboard container on 8082. Steel remains fully removed
> (re-probed CT201/CT333: no containers, no images, no routes).
> **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command > **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command
> from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** — > from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** —
> `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`, > `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,