Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).
Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.
- infrastructure/tailscale: node table rebuilt from tailscale status --json.
50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
re-verification. m93p records online (peer record chronically stale, third
occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
Seven job rows carried stale error claims for jobs since cleared to ok --
each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
failure notice was never delivered (deliveries.db has zero rows for
2026-10-30), so the incident row is the only trace. Job not re-enabled and
gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
tab bar is built from the service groups (renderTabs iterates state.groups),
so the settings.yaml layout block and its tab:/columns: keys are inert --
there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
amber; manager.py runs inside the homepage-manager container; and the "no
Traefik container" evidence was wrong -- the container exists but is
stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
occupy 443 stands. Same evidence error corrected in tailscale.md.
Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).
No secrets written -- provider names, key names and HTTP status codes only.
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.
Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)
Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).
Co-Authored-By: Hermes <hermes@local>
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).
STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.
New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.