- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
(CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
(running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
work-outlook (himalaya still login not supported), omniroute-free-deepseek
(good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
'lost its durable fire claim ownership' and left the tree dirty)
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.
New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
Ran wiki-maintain.sh (repo script). skills-index regenerated -> 302 skills / 68 categories.
Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt (761 files) and deployed to public/; hermes-wiki-serve.service restarted.
Live-vs-page drift diff (age scan alone missed all of this):
- infrastructure/tailscale.md: 1 state transition + 8 offline-age drifts + 2 dead Serve
routes. anthonys-mac-mini came back online (peer record Online:true, tailscale ping ->
pong via DERP(syd)). Offline ages refreshed +1d each (cloudreve 104d, elementsynapse
107d, freshrss-new 80d, octoagent 4d, octop 2d, omarchybook 15d, omarchymac 4d,
openmausbot 15d). Every peer IP still matches the live JSON; no peer missing, no
orphan row. /agent-estate->8769 and /synergy-prep->9123 are DEAD (no listener) while
still present in `tailscale serve status` - both now marked DEAD, matching
network-map.md. All other Serve targets have live listeners.
- infrastructure/docker-services.md: the 2026-11-05 "the seeder does not exist on any
host" finding was FALSE - a point-in-time sampling artefact. The worldmonitor-seeder-*
containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer
4h -> seeder-scheduled; worldmonitor-seed-fast.timer 15min -> seeder-fast), visible
only while a run is in flight. Both present at 2026-11-06 03:1x; neither at
2026-11-05 05:1x. The "no cron job" claim was also wrong (systemd timers, not cron) and
the "rename" narrative was wrong (one script, two container names). Corrected on the
page with strikethrough + dated correction. CT200's 61-container running set still
matches live set-wise. CT333 container renamed again: flamboyant_mahavira ->
nostalgic_keldysh.
- systems/scheduled-tasks.md: no drift. Live jobs.json reads 29 jobs / 26 enabled /
3 disabled, 1 non-clean last_status - exactly as recorded.
- current-state.md / index.md: false seeder claim and "every state matching" claim
marked superseded with dated corrections (SCHEMA rule 9), not silently rewritten.
- infrastructure/network-map.md: dates bumped, dead routes re-confirmed, peer recheck
note refreshed.
- log.md: appended (append-only) with the full stocktake record.
No secrets, tokens or keys written or quoted.
ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.
Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
matches across CT200/201/333/460); CT333 first container renamed again
(nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
vs 52 live IPv4 peers, no live peer missing, no orphan row, every
online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.
New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).
Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
- Add 2026-10 report: 9 orders, $301.04 total, $33.45 avg
- Regenerate 2026-07/08/09 with correct Australia/Perth timestamps
(Camofox browser TZ was hardcoded to America/Los_Angeles, making
Uber's DOM render receipt times 15h behind with AM/PM stripped)
- Totals unchanged for historical months; only timestamps corrected
- Preserve original 'created' frontmatter date on regeneration
- Fix cron job 066bb5191280 delivery target + port reference
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.
RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.
LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
times out); m93p peer record reads false for the 4th consecutive time while the
host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
(100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
worldmonitor-seeder-fast entry removed from the CT200 list, which is now
regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
46 archived. (301 is the unique (category, skill_name) count; find returns 302
files due to the known nested obsidian-git-sync alias duplicate.)
Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).
Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.
- infrastructure/tailscale: node table rebuilt from tailscale status --json.
50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
re-verification. m93p records online (peer record chronically stale, third
occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
Seven job rows carried stale error claims for jobs since cleared to ok --
each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
failure notice was never delivered (deliveries.db has zero rows for
2026-10-30), so the incident row is the only trace. Job not re-enabled and
gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
tab bar is built from the service groups (renderTabs iterates state.groups),
so the settings.yaml layout block and its tab:/columns: keys are inert --
there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
amber; manager.py runs inside the homepage-manager container; and the "no
Traefik container" evidence was wrong -- the container exists but is
stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
occupy 443 stands. Same evidence error corrected in tailscale.md.
Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).
No secrets written -- provider names, key names and HTTP status codes only.
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.
Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
noted. (303 SKILL.md files exist outside .archive/, but
productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
3 no-date. After the corrections below and a second script run: 0 stale /
107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.
Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
11 rows changed. Two were state transitions, not age drift:
* sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
* m93p is online despite a stale peer record (Online: false in the JSON) -
tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
Second occurrence of this pattern after 2026-10-26.
The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
ingress. No missing peers, no IP mismatches. Container identity re-confirmed
from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
CT601 still absent from both.
- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
"the connection credential itself was never re-authenticated" is falsified.
Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
(CT600) returns HTTP 200 - the connection has since been re-authenticated. The
same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
remediation was never reversed and remains correct.
- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
still on custom:omni-101 with routing aliases; both aliases HTTP 200;
ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
- still dead, different failure signature.
- systems/scheduled-tasks.md: the Known-Issues entry repeating the
"never re-authenticated" claim now carries the correction. Fleet figures
re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
3 disabled, 5 non-clean last_status).
- log.md / index.md: stocktake entry appended and index last-updated refreshed.
Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).
Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30
rollback disabled the browser-steel plugin and set browser.engine: camofox but
left browser.cloud_provider: steel in config.yaml. The split config was
inert-but-wrong, so it survived ~3 months of audits unnoticed.
Corrects pages that still described Steel as present, running or configured:
- systems/browser-backend.md (incl. gateway-restart env-var note)
- comparisons/steel-vs-camoufox.md, comparisons/camoufox-vs-chrome.md
- decisions/rollback-steel-to-camofox.md, decisions/index.md
- infrastructure/docker-services.md, infrastructure/tailscale.md, infrastructure/hosts.md
- systems/homepage-dashboard.md, systems/scheduled-tasks.md
- concepts/browser-automation.md, runbooks/recover-docker-service.md
- current-state.md, index.md, agents/nanobot.md
- log.md (dated entry; supersedes the 00:3x audit's 'endpoint dark' wording)
Also restores the Ignis compose row in infrastructure/docker-services.md, which
the 2026-10-11 revision dropped (verified: /home/hermes/docker/ignis/docker-compose.yml
exists and the ignis container is Up on CT460).
Co-Authored-By: Hermes <hermes@local>
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
Mark-don't-delete: the CT450 '3 commits behind' follow-up and the two
mnemosyne-vector-recall follow-ups are struck through with their verified
2026-10-26 resolution rather than removed.
- CT450 tracks origin/main (there is no origin/master on that remote) and is
0 commits behind.
- Coverage monitor b63ed5817212 is enabled, last_status ok, working coverage
96.8% — correctly has nothing to fire on.
- fastembed 0.8.1 now imports in both managed runtimes plus the legacy venv.
Vault at /home/hermes/wiki tracks 125 files under two conventions. This run
forward-synced all 22 touched-page copies (sha256-verified) and committed the
vault locally; it is local-only and was not pushed. Remaining drift measured by
hash sweep and recorded rather than silently mirrored:
top-level: match=25 stale=2 missing=80
wiki/ : match=34 stale=58 missing=15