Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.
Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)
Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).
Co-Authored-By: Hermes <hermes@local>
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
(4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
(springroll-browser was missing; omarchybook + openmausbot now offline; all offline
ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
- systems/skills-index.md: Regenerated from live skill tree (340 skills in 73 categories)
- raw/stocktake/freshness-20260921.txt: Freshness audit — 67 stale pages (≥21d), 5 fresh, 7 no-date
- infrastructure/tailscale.md: Verified against live tailscale serve status, reordered routes to match current state, updated verified_on to 2026-09-21
Replace the skills-only regeneration script with a general wiki-maintain.sh that:
- regenerates skills-index from live skill tree (330 skills / 26 categories)
- audits EVERY page for staleness vs its updated date (>21d flagged)
- rebuilds + redeploys the Quartz site
- writes a dated freshness report to raw/stocktake/
Also fix the tailscale.md page (it had drifted to only 4 routes; reality now has
11 CT460 routes incl /webui, /agent-estate, /synergy-prep, /evening-grid, /wiki
and the /wiki proxy was also missing from the atomic apply script — added there).
Cron wiki-stocktake-maintenance (1a8e90ea2271) now runs this daily 03:00 and
commits+pushes so the wiki cannot silently drift again.
The skills-index page had drifted badly (claimed 66 skills; 330 are actually
installed). Regenerate it from the live skill tree and add an automated
regeneration script + related cron so it can never go stale again.
- regenerate-skills-index.sh: full stock-take (inventory -> quartz build -> deploy)
- systems/skills-index.md: rebuilt with all 330 skills across 26 categories
- .gitignore: fix stray literal chars, ignore build artifacts