diff --git a/agents/nanobot.md b/agents/nanobot.md index 43d5f63..94072b6 100644 --- a/agents/nanobot.md +++ b/agents/nanobot.md @@ -3,8 +3,8 @@ title: Nanobot type: agent status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [agent, nanobot, homelab] sources: [live-probe, runtime-config] @@ -20,7 +20,7 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat - **Workspace:** `/home/nano/.nanobot/workspace` ## Current state -- **Status: ACTIVE.** Verified 2026-10-11: gateway process `nanobot-gateway` (PID 2259048) running and listening. +- **Status: ACTIVE.** Re-verified 2026-11-20: `nanobot-gateway` PID 1256104 (since 2026-11-19), `/health` → `status ok, ready true`, v0.4.1. (Prior check 2026-10-11: PID 2259048.) - **Gateway:** port `8765` (HTTP API + chat completions) — `LISTEN 0.0.0.0:8765` - **WebSocket bridge:** port `8766` — `LISTEN 0.0.0.0:8766` - **Serve API:** port `8900` was recorded as the Serve API; **not listening** in the 2026-10-11 `ss -ltn` check. Treat as unverified/possibly removed. @@ -38,22 +38,31 @@ Primary homelab AI agent running on `clawtest` (CT333). Handles agent orchestrat Other listeners on CT333 include 8787, 8880, 8899, 8999, 5050, 8080 and the Tailscale 443. *(Port 3000 was published by the `steel-browser` container, which was removed 2026-10-29.)* ## Docker on CT333 (verified 2026-10-11; Steel container removed 2026-10-29) -`gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `tender_brattain`, plus healthy `community-spark` and `quirky_darwin`. +~~`tender_brattain`, plus healthy `community-spark` and `quirky_darwin`~~ → **2026-11-20 live list:** `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, `edge-tts`, `onecli-postgres-1`, `ecstatic_swartz` (random-name successor of `tender_brattain`), healthy `community-spark`; `quirky_darwin` removed; stopped `qdrant` present (Exited 5 months ago). **Steel is gone from CT333 too.** The `steel-browser` container (image `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image were removed on **2026-10-29**. It was already orphaned — its last CDP navigation was 2026-10-20, it crashed on 2026-10-22, it received no requests after 2026-10-21, and it had no Tailscale Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on native_webdriver at `127.0.0.1:9515`), and no reference from Nanobot's browser harness. → [[comparisons/steel-vs-camoufox]] ## Cron jobs -Unverified against live Nanobot scheduler on 2026-10-11. Previously recorded: -- `auto-upgrade-freellmapi-safeguard` — daily at 03:30 AWST -- `morning-quota-report` — daily at 07:30 AWST -- `daily-quota-730am` — daily at 07:30 AWST -- `vault-daily-seed` — daily at 08:00 AWST -- `daily-agent-briefing` — daily at 08:15 AWST -- `rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST -- `daily-quota-7pm` — daily at 19:00 AWST -- `evening-quota-report` — daily at 21:00 AWST -- `heartbeat` — every 30m -- `dream` — every 2h (memory consolidation) +**Verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` — **11 jobs, 8 enabled** +(all schedules Australia/Perth): + +| Job | Schedule | Enabled | +|---|---|---| +| `auto-upgrade-freellmapi-safeguard` | daily 03:30 | ✅ | +| `nightly-skills-export-sync` | daily 04:00 | ✅ | +| `morning-news-podcast` | daily 07:00 | ✅ | +| `daily-quota-730am` | daily 07:30 | ✅ | +| `morning-quota-report` | daily 07:30 | ❌ disabled | +| `vault-daily-seed` | daily 08:00 | ✅ | +| `daily-agent-briefing` | daily 08:15 | ❌ disabled (2026-10-30) | +| `daily-quota-7pm` | daily 19:00 | ✅ | +| `evening-quota-report` | daily 21:00 | ❌ disabled | +| `heartbeat` | every 30m | ✅ | +| `dream` | every 2h (memory consolidation) | ✅ | + +~~`rss-check-proxmox-selfhst` — Mon/Thu at 09:00 AWST~~ **No longer in the scheduler** (removed at +some point before 2026-11-20; the enabled quota-notification pair `daily-quota-730am` / +`daily-quota-7pm` remains enabled, per Anthony's standing preference). ## Skills Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli-app-exa`, `playwright`, `system-prompts-collection`, `agent-skill-retrieval`, `enhanced-search`, `api-gateway`, `debugging-and-error-recovery`, `mnemosyne-memory-hooks`, `proxmox-shell-fallback`, `agentmail`, `langfuse`, `install-and-verify-agent-skills`, `verify-system-claims`, `skill-interceptor`, `tool-call-failure-recovery`, `using-agent-skills`, `api-vault`, `repair-memory`, `patch-config-relative-paths`, `long-task-management`, `nanobot-full-restore-backup`, `context-diet`, `frontend-design`. @@ -73,5 +82,21 @@ Active skills include: `agent-reach`, `verify-nanobot-config-against-docs`, `cli - [[runbooks/restart-hermes]] - [[runbooks/nanobot-restart]] +> **Re-verified 2026-11-20 (stocktake, live probes on CT333).** Gateway healthy: +> `GET :8765/health` → `{"status":"ok","ready":true,"websocket":"running","version":"0.4.1"}`; +> `nanobot-gateway` PID **1256104** (started 2026-11-19), ports 8765 + 8766 listening, 8900 still +> **not listening**. Tailscale IP confirmed `100.122.189.81`. The mnemosyne MCP, chroma_query_api, +> agentmail-daemon and readlater (:3210) processes all still run. +> +> **Docker drift corrected:** live `docker ps -a` shows `ecstatic_swartz` (Up 19h) where this page +> listed `tender_brattain` — Docker regenerates random names for compose-less containers, so this is +> a rename-class drift, not a new service. `quirky_darwin` is **absent even with `-a`** (removed, not +> stopped). `community-spark`, `gotify`, `readlater`, `readlater-extraction-worker`, `kokoro-tts`, +> `edge-tts`, `onecli-postgres-1` all Up ~9 days and healthy where applicable; a stopped `qdrant` +> (Exited 5 months ago) also exists. No steel container or image remains; nothing on :3000. +> +> **Cron verified live 2026-11-20** from `/home/nano/.nanobot/workspace/cron/jobs.json` (the +> `~/.nanobot/cron/` dir is empty — the scheduler file lives under `workspace/cron/`): +> **11 jobs, 8 enabled** — see the rewritten Cron jobs section above. ## Related - [[comparisons/hermes-vs-nanobot]] diff --git a/comparisons/camoufox-vs-chrome.md b/comparisons/camoufox-vs-chrome.md index 7e8e8cc..fc7ae03 100644 --- a/comparisons/camoufox-vs-chrome.md +++ b/comparisons/camoufox-vs-chrome.md @@ -3,8 +3,8 @@ title: Camoufox vs Chrome type: comparison status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: medium tags: [comparison, browser] sources: [raw/configs/hermes-config-sanitized.txt] @@ -46,6 +46,16 @@ Compare the two browser-automation options for Hermes's browser tool. - Runs headed under Xvfb rather than a true headless mode, per [[systems/headless-server]] — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing. - `cua-driver` bridges Hermes's computer-use tool calls to the browser. +> **Re-verified 2026-11-20 (stocktake).** Live config unchanged: `cloud_provider: camofox`, +> `camera_provider: camofox`, `engine: auto`, `headed: true`, `inactivity_timeout: 450`, +> `use_gateway: false`, `cdp_url: ""`, `camofox.managed_persistence: true`, `user_id: anthony`. +> `cua-driver` still **v0.28.2**; `cua-driver serve` + `camoufox-ws-server.mjs` + a `camoufox-bin` +> tree all running on CT460 (under Xvfb `:99` 1600×1200 + fluxbox). +> **Correction:** ~~Camofox's REST port 9377 is not listening on CT460 — the service runs on CT450~~ — +> `:9377` **is listening on CT460** (`node`, `/health` → `ok`, engine `camoufox`); the service is the +> local `systemd --user` unit `camofox-browser.service` (active), per [[systems/browser-backend]]'s +> 2026-10-13 correction. The CT450 attribution is dead. The ws-server also listens on `:9222` (CDP). +> No Chrome/Chromium process found in this probe either. ### Chrome (not found in use) - No `google-chrome`, `chromium`, or CDP-only launch flags found in the sanitized config or running process list during this audit. - Included in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn't captured anywhere; worth a [[decisions/index]] entry if anyone remembers why Camoufox was chosen over stock Chrome. diff --git a/comparisons/steel-vs-camoufox.md b/comparisons/steel-vs-camoufox.md index 30720d0..d525262 100644 --- a/comparisons/steel-vs-camoufox.md +++ b/comparisons/steel-vs-camoufox.md @@ -3,8 +3,8 @@ title: Steel Browser vs Camoufox type: comparison status: active created: 2026-07-29 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [comparison, browser, steel, camofox, decision] sources: [live-probe, session-history] @@ -77,6 +77,13 @@ the CT201 and CT333 deployments torn down — so returning to Steel means redepl - **Reversibility:** the compose file and Dockerfile are archived on CT201 under `/root/_archived-steel/`, plus `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. +> **Re-verified 2026-11-20 (stocktake).** Steel-removal claims re-probed live: CT201 has **no** steel +> container or image, nothing on :3000/:5173/:9223, Serve routes are only `/` → 8085 and `/mgr` → 8084 +> on :443 **and** :8443; `/root/_archived-steel/` holds `docker-compose.yml`, `Dockerfile.api`, +> `steel-cache-archive-20261029.tgz`. CT333 likewise has no steel container/image and nothing on :3000. +> `~/.hermes/plugins/browser-steel/` absent. Camofox healthy on CT460: `:9377/health` → `ok` +> (idle — `browserConnected:false` at probe time, no active session), unit active, config +> `cloud_provider: camofox`. All removal claims **hold**. ## Related - [[systems/browser-backend]] - [[concepts/browser-automation]] diff --git a/concepts/browser-automation.md b/concepts/browser-automation.md index 21566e0..bbb2719 100644 --- a/concepts/browser-automation.md +++ b/concepts/browser-automation.md @@ -3,8 +3,8 @@ title: Browser Automation type: concept status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [concept, browser, camofox, steel] sources: [live-probe, session-history] @@ -33,6 +33,10 @@ Steel Browser (Chromium 140 on CT201) was briefly the primary backend. It offere **Lesson:** A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The `browser-steel` plugin **was deleted on 2026-10-29** and the Steel deployments on CT201 and CT333 were torn down, so there is no Steel fallback left to fall back to — Camofox is the only backend. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]]. +> **Re-verified 2026-11-20 (stocktake).** Camofox service active on CT460 (`systemctl --user`), +> `:9377/health` → `ok`; Xvfb `:99` at 1600×1200×24 with fluxbox, websockify on 127.0.0.1:6080. +> Config `cloud_provider: camofox` / `camera_provider: camofox` confirmed. No Steel anywhere +> (CT201/CT333 containers, images, routes, plugin all re-probed absent). ## Related - [[systems/browser-backend]] - [[comparisons/steel-vs-camoufox]] diff --git a/decisions/index.md b/decisions/index.md index 3c89f62..a96620d 100644 --- a/decisions/index.md +++ b/decisions/index.md @@ -3,8 +3,8 @@ title: Decisions type: collection status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [decisions] --- @@ -31,6 +31,10 @@ Stable strategic and config decisions, with evidence and status. > plugin deleted, CT201 and CT333 deployments torn down, Serve routes deleted. Camofox is the sole > backend. +> **Status sweep 2026-11-20:** all three decisions still `status: active` and re-verified against live +> state — `keep-openrouter-as-fallback` and `persistent-shell-enabled` were re-verified 2026-11-11 +> (`terminal.persistent_shell: true` re-confirmed live today); `rollback-steel-to-camofox`'s +> 2026-10-29 update re-probed today: `cloud_provider: camofox`, Steel absent estate-wide. ## History - **Verified:** Decision pages initialized during wiki setup on 2026-07-22 - **Inferred:** Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing diff --git a/decisions/rollback-steel-to-camofox.md b/decisions/rollback-steel-to-camofox.md index 306bede..99c6b63 100644 --- a/decisions/rollback-steel-to-camofox.md +++ b/decisions/rollback-steel-to-camofox.md @@ -3,8 +3,8 @@ title: Rollback Steel to Camoufox type: decision status: active created: 2026-08-02 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [decision, browser, steel, camofox, rollback] sources: [session-history, live-config] @@ -50,7 +50,7 @@ Steel Browser was deployed on CT201 on 2026-07-29 as the primary browser automat > restoration was needed or wanted. ## Rationale -1. **Reliability:** Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. +1. **Reliability:** Camoufox runs on the same host as Hermes (~~CT450~~ **CT460** — host attribution corrected 2026-11-20; the service is the local `camofox-browser.service` on the production Hermes host) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues. 2. **Auth persistence:** Camoufox's persistent profile `anthony` is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports. 3. **Failure mode:** The `opencode-go/deepseek-v4-flash` empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem. 4. **DeepSeek 0731 compatibility:** The new model's stricter tool name validation (`^[a-zA-Z0-9_-]+$`) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain. @@ -103,6 +103,14 @@ removed**, so the `plugin_steel` toolset is no longer selectable. The decision itself is unchanged — Camofox is the backend — but the "Steel is still there if you need it" assumption is dead. Detail: [[systems/browser-backend]], [[comparisons/steel-vs-camoufox]]. +> **Re-verified 2026-11-20 (stocktake).** Every removal claim in the 2026-10-29 update re-probed: +> config `cloud_provider/camera_provider: camofox`, `engine: auto`; plugin dir absent; CT201 steel +> containers/images/network/project dir gone, Serve only `/`→8085 + `/mgr`→8084 on :443/:8443, +> nothing on 3000/5173/9223; CT333 steel gone, :3000 closed; `/root/_archived-steel/` archive intact +> (compose, Dockerfile.api, cache tgz). Camofox `:9377/health` → `ok` (idle). **Note:** the Rationale's +> "Camoufox runs on the same host as Hermes (CT450)" reflects the July-era attribution — the service +> runs on **CT460** (the production Hermes host), corrected in [[systems/browser-backend]] 2026-10-13. +> The decision and its consequences are unchanged. ## Related - [[comparisons/steel-vs-camoufox]] - [[systems/browser-backend]] diff --git a/incidents/2026-10-01-cliproxy-connection-auth-expired.md b/incidents/2026-10-01-cliproxy-connection-auth-expired.md index 1df33ce..a4401b1 100644 --- a/incidents/2026-10-01-cliproxy-connection-auth-expired.md +++ b/incidents/2026-10-01-cliproxy-connection-auth-expired.md @@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry type: incident status: resolved created: 2026-10-02 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [cron, omniroute, provider, auth, incident, resolved] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040, cron-session-cron_79f9bb41ecf5_20261004_180050] @@ -130,6 +130,12 @@ done. - **Catalog-missing model** (HTTP 400, jobs `fad5a86979d4` / `0e753e9720fd`) — the model name is absent from the provider's live catalog. +> **Re-verified 2026-11-20 (stocktake).** The 2026-10-29 correction still holds, re-probed live: +> `cliprox/terra` on `custom:Omniroute:20128` (CT600) → **HTTP 200** (resolves to `…/terra`); +> on `custom:omni-101` (CT101) → **HTTP 401** `No active credentials for provider: cliprox.` — +> per-node credentials unchanged. `jobs.json`: **zero** jobs reference `cliprox` anywhere; all three +> originally-exposed jobs (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`) remain on +> `custom:omni-101` / `free-stack`, `failure_streak: 0`, `last_status: ok`. Resolution stands. ## Related - [[incidents/2026-10-05-dead-model-pin-repin]] diff --git a/incidents/2026-10-05-dead-model-pin-repin.md b/incidents/2026-10-05-dead-model-pin-repin.md index 38e92f7..d1cf10c 100644 --- a/incidents/2026-10-05-dead-model-pin-repin.md +++ b/incidents/2026-10-05-dead-model-pin-repin.md @@ -3,8 +3,8 @@ title: Seven Cron Jobs on Dead Model Pins Repinned to free-stack / smart-route type: incident status: resolved created: 2026-10-06 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 confidence: high tags: [cron, model-pin, omniroute, incident, resolved] sources: [live-jobs-json, live-omniroute-chat-completions-probe, cron-executions-db, cron-incidents-db, session-20260810_122140_d687279c] @@ -119,6 +119,17 @@ state rather than left on trust. - The daily stocktake (`1a8e90ea2271`) had been blocked since 2026-09-25; with the repin it can run again at its 03:00 schedule, so `wiki-maintain.sh` no longer needs a manual run each day. +> **Re-verified 2026-11-20 (stocktake).** All seven jobs still on `custom:omni-101` with the repinned +> aliases (`free-stack` ×6, `smart-route` ×1 for `dca8482e4f76`); **every one now reads +> `last_status: ok`, `failure_streak: 0`** — the stale pre-repin `error` statuses noted in Residual +> have cleared. `free-stack` and `smart-route` probed live on **both** OmniRoute nodes → HTTP 200 +> (CT600: free-stack → `openrouter/dots-studio/dots-3-note-preview:free`; CT101: both → +> `qwen/qwen4-max:free`). `cliprox/terra`: 200 on CT600 / 401 on CT101 (unchanged). +> **Error-class drift again on `command-code/gpt-5.6-terra`:** now **HTTP 403 `error code: 1010`** +> (Cloudflare access-denied signature) where 2026-10-29 saw 401 and the incident recorded 403 +> `MODEL_NOT_IN_PLAN`. The pin remains dead; only the rejection layer differs. Fleet is now +> **29 jobs / 26 enabled** (was 30/27 at the fix) — composition changes are tracked in +> [[systems/scheduled-tasks]], not here. ## Related - [[systems/scheduled-tasks]] diff --git a/log.md b/log.md index 2e92e3b..ccdd404 100644 --- a/log.md +++ b/log.md @@ -1877,3 +1877,16 @@ recorded here so the next audit does not re-litigate it. - **Mnemosyne consolidation is still zero-item:** newest `consolidation_log` row **`2026-11-19T16:20:14+00:00`** (= 2026-11-20 00:20 AWST, ~31 min cadence) with `items_consolidated: 0` and `summary_preview: "No-op consolidation cycle — timestamp updated for watchdog"`; **199 of the last 200 cycles are zero-item** (was **120** on the 11-17 read). The zero-item-streak threshold remains **proposed, not applied**. - **Freshness at publish time** (`raw/stocktake/freshness-20261120.txt`): **9 stale / 127 fresh / 3 no-date of 139 pages**. All nine carry the *same* `updated: 2026-10-29` — one 22-day bulk cohort, not nine findings: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. **Not re-verified by this audit** — the live-vs-page re-verification sweep belongs to the 03:00 stocktake (pattern 23), so the cohort is recorded here rather than silently inherited. Five further pages sit at exactly 21d and become the 11-21 stale list: `incidents/2026-10-27-job-radar-budget-exhausted.md`, `incidents/2026-10-28-job-radar-stage1-iteration-cap.md`, `incidents/2026-10-29-conduit-voice-silence.md`, `incidents/2026-10-30-m93p-nfs-mount-loss-lxc-boot-failure.md`, `systems/homepage-dashboard.md`. - **Published (patterns 13/30):** `wiki-maintain.sh` regenerated `systems/skills-index.md` (**299 skills / 67 categories**) and rebuilt Quartz — 139 pages parsed, 867 files emitted to `public/`, `hermes-wiki-serve.service` restarted (`systemctl --user is-active` → `active`, listener on `:9120`). Live check after the rebuild: `index`, `current-state`, `systems/mnemosyne`, `systems/scheduled-tasks`, `systems/skills-index` and all three incident pages return **200** on `http://127.0.0.1:9120/`. Both vault conventions synced from the authoring tree with per-file `sha256` matches (8 files × 2 conventions; freshness report included), committed locally — the vault repo has no remote. + +## 2026-11-20 03:0x — Stocktake (cron 1a8e90ea2271): 9-page stale cohort re-verified live + +- Ran `wiki-maintain.sh` (repo script): skills-index regenerated (**299 skills / 67 categories**), freshness report `raw/stocktake/freshness-20261120.txt` → **9 stale / 127 fresh / 3 no-date of 139**. All nine stale pages were the single 2026-10-29 cohort (22d) flagged by the 01:13 audit — treated as one verification sweep (pattern 23), not nine investigations. +- **Live probes:** `~/.hermes/config.yaml` browser block; `ss -ltnp` + process list + `cua-driver --version` on CT460; `camofox-browser.service` (`systemctl --user` → active) + `:9377/health` → ok; CT333 via PVE (`ss`, `ps`, `docker ps -a`, `tailscale ip -4`, gateway `/health`, `workspace/cron/jobs.json`); CT201 via PVE (`docker ps -a`, images, Serve status, `_archived-steel/`, `:8085` probe); OmniRoute chat-completions on both nodes (`cliprox/terra`, `free-stack`, `smart-route`); `command-code/gpt-5.6-terra`; `jobs.json` fleet read. +- **All 9 pages re-verified** (`updated`/`verified_on` → 2026-11-20) with dated notes: `agents/nanobot.md`, `comparisons/camoufox-vs-chrome.md`, `comparisons/steel-vs-camoufox.md`, `concepts/browser-automation.md`, `decisions/index.md`, `decisions/rollback-steel-to-camofox.md`, `incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `incidents/2026-10-05-dead-model-pin-repin.md`, `runbooks/recover-docker-service.md`. +- **Corrections (claims disproved or drifted):** + - `agents/nanobot.md`: Docker list drifted — `tender_brattain` → `ecstatic_swartz` (random-name rename class), `quirky_darwin` removed (absent with `-a`), stopped `qdrant` present. Gateway PID now 1256104 (v0.4.1, `/health` ok). **Cron section rewritten from live scheduler**: 11 jobs / 8 enabled; `morning-quota-report`, `daily-agent-briefing`, `evening-quota-report` disabled; `rss-check-proxmox-selfhst` gone; new `nightly-skills-export-sync` (04:00) and `morning-news-podcast` (07:00). Quota pair 07:30/19:00 remains enabled. Scheduler file is `workspace/cron/jobs.json` (the `~/.nanobot/cron/` dir is empty). + - `comparisons/camoufox-vs-chrome.md`: struck the stale "9377 not listening on CT460 / service on CT450" claim — `:9377` **is** listening on CT460 (local `camofox-browser.service`, active); ws-server also on `:9222`. + - `decisions/rollback-steel-to-camofox.md`: Rationale's "(CT450)" host attribution corrected to **CT460** with strikethrough. + - `incidents/2026-10-05-dead-model-pin-repin.md`: residual "four jobs still carry pre-repin error status" has **cleared** — all seven now `last_status: ok`, streak 0. `command-code/gpt-5.6-terra` error class drifted **again**: now 403 `error code: 1010` (Cloudflare) vs 401 on 10-29 vs 403 MODEL_NOT_IN_PLAN originally — pin still dead, rejection layer differs. Fleet now 29 jobs / 26 enabled (was 30/27). +- **Claims that held:** Steel fully removed (CT201 + CT333: no containers/images/routes; `_archived-steel/` intact); `cliprox/terra` 200 on CT600 / 401 on CT101 (per-node credentials, unchanged since 10-29); zero jobs on `cliprox`; `free-stack`/`smart-route` 200 on both nodes; docker.sock permission-denied for `hermes` on CT460; CT201 homepage `:8085` still 502 (degradation persists). +- Not re-probed: the five pages at exactly 21d (job-radar ×2, conduit-voice, m93p-nfs, homepage-dashboard) — they become the 11-21 stale list; homepage's core claim was incidentally re-confirmed (502 persists) but the page itself was left for its own cycle. Pre-existing dangling wikilinks in `agents/nanobot.md` (`systems/gitea`, `systems/chromadb`, `incidents/nanobot-degraded-2026-08-07`, `runbooks/nanobot-restart`) confirmed present before this edit — flagged, not introduced. diff --git a/raw/stocktake/freshness-20261120.txt b/raw/stocktake/freshness-20261120.txt index 4863c3e..88214c4 100644 --- a/raw/stocktake/freshness-20261120.txt +++ b/raw/stocktake/freshness-20261120.txt @@ -1,37 +1,34 @@ # Wiki freshness audit — 2026-11-20 Threshold: >21 days since `updated` -## STALE (9) +## STALE (0) -- [ 22d] agents/nanobot.md (updated 2026-10-29) -- [ 22d] comparisons/camoufox-vs-chrome.md (updated 2026-10-29) -- [ 22d] comparisons/steel-vs-camoufox.md (updated 2026-10-29) -- [ 22d] concepts/browser-automation.md (updated 2026-10-29) -- [ 22d] decisions/index.md (updated 2026-10-29) -- [ 22d] decisions/rollback-steel-to-camofox.md (updated 2026-10-29) -- [ 22d] incidents/2026-10-01-cliproxy-connection-auth-expired.md (updated 2026-10-29) -- [ 22d] incidents/2026-10-05-dead-model-pin-repin.md (updated 2026-10-29) -- [ 22d] runbooks/recover-docker-service.md (updated 2026-10-29) -## FRESH (127) +## FRESH (136) - [ 9d] CONTRIBUTING.md - [ 12d] agents/hermes-chaos.md - [ 12d] agents/hermes-production.md - [ 9d] agents/index.md +- [ 0d] agents/nanobot.md - [ 9d] agents/open-claw.md - [ 16d] agents/qwenpaw.md - [ 9d] agents/zeroclaw.md +- [ 0d] comparisons/camoufox-vs-chrome.md - [ 9d] comparisons/hermes-vs-nanobot.md - [ 9d] comparisons/memory-backends.md - [ 9d] comparisons/model-routers.md +- [ 0d] comparisons/steel-vs-camoufox.md - [ 9d] concepts/agent-memory.md +- [ 0d] concepts/browser-automation.md - [ 9d] concepts/model-routing.md - [ 9d] concepts/persistent-shells.md - [ 9d] concepts/tool-calling.md - [ 0d] current-state.md +- [ 0d] decisions/index.md - [ 9d] decisions/keep-openrouter-as-fallback.md - [ 9d] decisions/persistent-shell-enabled.md +- [ 0d] decisions/rollback-steel-to-camofox.md - [ 9d] experiments/index.md - [ 8d] incidents/2026-07-29-mnemosyne-relay.md - [ 9d] incidents/2026-08-01-mnemosyne-capture-failure.md @@ -43,8 +40,10 @@ Threshold: >21 days since `updated` - [ 9d] incidents/2026-08-29-gmail-read-state.md - [ 8d] incidents/2026-09-02-sftpgo-outage.md - [ 16d] incidents/2026-09-28-hermes-managed-runtime-deps.md +- [ 0d] incidents/2026-10-01-cliproxy-connection-auth-expired.md - [ 2d] incidents/2026-10-03-ct450-hermes-update.md - [ 2d] incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md +- [ 0d] incidents/2026-10-05-dead-model-pin-repin.md - [ 5d] incidents/2026-10-09-cron-provider-timeout-storm.md - [ 19d] incidents/2026-10-09-memory-recall-relevance.md - [ 14d] incidents/2026-10-14-omniroute-node-connection-refused-storm.md @@ -107,6 +106,7 @@ Threshold: >21 days since `updated` - [ 13d] runbooks/job-radar.md - [ 9d] runbooks/multiplexer-setup.md - [ 2d] runbooks/provider-health.md +- [ 0d] runbooks/recover-docker-service.md - [ 9d] runbooks/restart-browser.md - [ 9d] runbooks/restart-hermes.md - [ 9d] runbooks/restore-openrouter.md @@ -149,4 +149,4 @@ Threshold: >21 days since `updated` - experiments/active/example.md - log.md -TOTAL: 9 stale, 127 fresh, 3 no-date of 139 pages \ No newline at end of file +TOTAL: 0 stale, 136 fresh, 3 no-date of 139 pages \ No newline at end of file diff --git a/runbooks/recover-docker-service.md b/runbooks/recover-docker-service.md index 1c106a1..f1694c9 100644 --- a/runbooks/recover-docker-service.md +++ b/runbooks/recover-docker-service.md @@ -3,8 +3,8 @@ title: Recover a Docker service type: runbook status: active created: 2026-07-22 -updated: 2026-10-29 -verified_on: 2026-10-29 +updated: 2026-11-20 +verified_on: 2026-11-20 tags: [runbook, docker, recovery] sources: [] confidence: high @@ -20,6 +20,11 @@ Container shows healthy after recovery probe reboots. ## Last successfully used 2026-07-22 +> **Re-verified 2026-11-20 (stocktake, procedure only).** `docker ps` as `hermes` on CT460 still +> returns permission denied (`/var/run/docker.sock` is `root:docker 0660`) — sudo or SSH to the +> owning host remains required. CT201 homepage degradation **persists**: `:8085` answers **502** +> locally, `homepage-manager` up but no dashboard container on 8082. Steel remains fully removed +> (re-probed CT201/CT333: no containers, no images, no routes). > **Verified 2026-10-20 (procedure, not a live recovery).** Before running any `docker` command > from CT460, note that the **`hermes` user has no access to `/var/run/docker.sock`** — > `docker ps` returns `permission denied while trying to connect to the docker API`. Use `sudo`,