Ran wiki-maintain.sh (repo script). skills-index regenerated -> 302 skills / 68 categories.
Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt (761 files) and deployed to public/; hermes-wiki-serve.service restarted.
Live-vs-page drift diff (age scan alone missed all of this):
- infrastructure/tailscale.md: 1 state transition + 8 offline-age drifts + 2 dead Serve
routes. anthonys-mac-mini came back online (peer record Online:true, tailscale ping ->
pong via DERP(syd)). Offline ages refreshed +1d each (cloudreve 104d, elementsynapse
107d, freshrss-new 80d, octoagent 4d, octop 2d, omarchybook 15d, omarchymac 4d,
openmausbot 15d). Every peer IP still matches the live JSON; no peer missing, no
orphan row. /agent-estate->8769 and /synergy-prep->9123 are DEAD (no listener) while
still present in `tailscale serve status` - both now marked DEAD, matching
network-map.md. All other Serve targets have live listeners.
- infrastructure/docker-services.md: the 2026-11-05 "the seeder does not exist on any
host" finding was FALSE - a point-in-time sampling artefact. The worldmonitor-seeder-*
containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer
4h -> seeder-scheduled; worldmonitor-seed-fast.timer 15min -> seeder-fast), visible
only while a run is in flight. Both present at 2026-11-06 03:1x; neither at
2026-11-05 05:1x. The "no cron job" claim was also wrong (systemd timers, not cron) and
the "rename" narrative was wrong (one script, two container names). Corrected on the
page with strikethrough + dated correction. CT200's 61-container running set still
matches live set-wise. CT333 container renamed again: flamboyant_mahavira ->
nostalgic_keldysh.
- systems/scheduled-tasks.md: no drift. Live jobs.json reads 29 jobs / 26 enabled /
3 disabled, 1 non-clean last_status - exactly as recorded.
- current-state.md / index.md: false seeder claim and "every state matching" claim
marked superseded with dated corrections (SCHEMA rule 9), not silently rewritten.
- infrastructure/network-map.md: dates bumped, dead routes re-confirmed, peer recheck
note refreshed.
- log.md: appended (append-only) with the full stocktake record.
No secrets, tokens or keys written or quoted.
ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.
Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
matches across CT200/201/333/460); CT333 first container renamed again
(nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
vs 52 live IPv4 peers, no live peer missing, no orphan row, every
online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.
New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).
Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.
RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.
LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
times out); m93p peer record reads false for the 4th consecutive time while the
host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
(100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
worldmonitor-seeder-fast entry removed from the CT200 list, which is now
regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
46 archived. (301 is the unique (category, skill_name) count; find returns 302
files due to the known nested obsidian-git-sync alias duplicate.)
Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).
Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.
- infrastructure/tailscale: node table rebuilt from tailscale status --json.
50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
re-verification. m93p records online (peer record chronically stale, third
occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
Seven job rows carried stale error claims for jobs since cleared to ok --
each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
failure notice was never delivered (deliveries.db has zero rows for
2026-10-30), so the incident row is the only trace. Job not re-enabled and
gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
tab bar is built from the service groups (renderTabs iterates state.groups),
so the settings.yaml layout block and its tab:/columns: keys are inert --
there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
amber; manager.py runs inside the homepage-manager container; and the "no
Traefik container" evidence was wrong -- the container exists but is
stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
occupy 443 stands. Same evidence error corrected in tailscale.md.
Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).
No secrets written -- provider names, key names and HTTP status codes only.
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.
Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
noted. (303 SKILL.md files exist outside .archive/, but
productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
3 no-date. After the corrections below and a second script run: 0 stale /
107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.
Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
11 rows changed. Two were state transitions, not age drift:
* sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
* m93p is online despite a stale peer record (Online: false in the JSON) -
tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
Second occurrence of this pattern after 2026-10-26.
The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
ingress. No missing peers, no IP mismatches. Container identity re-confirmed
from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
CT601 still absent from both.
- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
"the connection credential itself was never re-authenticated" is falsified.
Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
(CT600) returns HTTP 200 - the connection has since been re-authenticated. The
same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
remediation was never reversed and remains correct.
- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
still on custom:omni-101 with routing aliases; both aliases HTTP 200;
ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
- still dead, different failure signature.
- systems/scheduled-tasks.md: the Known-Issues entry repeating the
"never re-authenticated" claim now carries the correction. Fleet figures
re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
3 disabled, 5 non-clean last_status).
- log.md / index.md: stocktake entry appended and index last-updated refreshed.
Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).
Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.
Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)
Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).
Co-Authored-By: Hermes <hermes@local>
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
(4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
(springroll-browser was missing; omarchybook + openmausbot now offline; all offline
ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
Drift found and corrected against live sources (not guessed):
- OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live);
CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted.
Hermes custom:omni resolves to omniroute -> CT600, not CT601.
- Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401),
registered as custom provider omni-101 (5384 models, default smart-route,
carries skills_hub/free-stack). Not in pct list; identity unconfirmed.
- Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and
CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults).
Both return HTTP 200. Page now distinguishes them.
- Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html);
undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected
(8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row).
- Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy
(both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed.
VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted.
- Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd
nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900
no longer listening; "Serve API" claim flagged unverified.
- Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----,
hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers),
CT333 (9 containers), CT460 via sudo.
- Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d.
Flagged, not investigated (no root cause):
- mycompose-db-1 on CT200 is in Restarting (1) - crash-looping.
62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions,
comparisons, most runbooks) - not probed against a live source, so reported not faked.
Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed.
Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
- systems/skills-index.md: Regenerated from live skill tree (340 skills in 73 categories)
- raw/stocktake/freshness-20260921.txt: Freshness audit — 67 stale pages (≥21d), 5 fresh, 7 no-date
- infrastructure/tailscale.md: Verified against live tailscale serve status, reordered routes to match current state, updated verified_on to 2026-09-21
Replace the skills-only regeneration script with a general wiki-maintain.sh that:
- regenerates skills-index from live skill tree (330 skills / 26 categories)
- audits EVERY page for staleness vs its updated date (>21d flagged)
- rebuilds + redeploys the Quartz site
- writes a dated freshness report to raw/stocktake/
Also fix the tailscale.md page (it had drifted to only 4 routes; reality now has
11 CT460 routes incl /webui, /agent-estate, /synergy-prep, /evening-grid, /wiki
and the /wiki proxy was also missing from the atomic apply script — added there).
Cron wiki-stocktake-maintenance (1a8e90ea2271) now runs this daily 03:00 and
commits+pushes so the wiki cannot silently drift again.