Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.
- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
either hypervisor; CT333 verified running; both hypervisors verified; config
33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
fallback chain rewritten from live config (first entry is the dead nous
credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
0 behind origin/master; default model smart-route on custom:omni-101; cron
"11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
online; every offline age refreshed; omni row corrected to "CT601 removed
from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
(80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
(provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
Current-state sections recording current versions and embedding coverage.
All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
(4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
(springroll-browser was missing; omarchybook + openmausbot now offline; all offline
ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
- systems/skills-index.md: Regenerated from live skill tree (340 skills in 73 categories)
- raw/stocktake/freshness-20260921.txt: Freshness audit — 67 stale pages (≥21d), 5 fresh, 7 no-date
- infrastructure/tailscale.md: Verified against live tailscale serve status, reordered routes to match current state, updated verified_on to 2026-09-21
Replace the skills-only regeneration script with a general wiki-maintain.sh that:
- regenerates skills-index from live skill tree (330 skills / 26 categories)
- audits EVERY page for staleness vs its updated date (>21d flagged)
- rebuilds + redeploys the Quartz site
- writes a dated freshness report to raw/stocktake/
Also fix the tailscale.md page (it had drifted to only 4 routes; reality now has
11 CT460 routes incl /webui, /agent-estate, /synergy-prep, /evening-grid, /wiki
and the /wiki proxy was also missing from the atomic apply script — added there).
Cron wiki-stocktake-maintenance (1a8e90ea2271) now runs this daily 03:00 and
commits+pushes so the wiki cannot silently drift again.