- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
(CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
(running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
work-outlook (himalaya still login not supported), omniroute-free-deepseek
(good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
- Wire host-scoped Gitea credential (git credential approve); origin was HTTPS with no userinfo
- Strip 1.76GB checkpoints/ from the unpushed range via git filter-repo --partial (fast-forward preserved)
- Untrack checkpoints/ at the tip; push set ~2GB -> 264.8MB
- Rewrite backup.sh: exit non-zero on failure, stamp .last_backup_ok only after remote verification,
decide push by remote-vs-local, guard ls-remote with || true, GitHub orphan snapshot via commit-tree,
honest AgentMail/Telegram delivery reporting
- Verified live: exit 0, origin/main == local HEAD, 0 unpushed, Gitea + GitHub APIs confirmed
- Incident page -> status: resolved; backups.md -> RESOLVED; scheduled-tasks job row updated
- Also commits the uncommitted stocktake edits (the 03:00 stocktake cron failed with
'lost its durable fire claim ownership' and left the tree dirty)
incidents/2026-10-14-omniroute-node-connection-refused-storm.md was at exactly 21d and
would have tripped the threshold on the next run. Its residual claim ("omniroute-1
answers again") was re-probed live and still holds: TCP connect to 100.78.63.20:20128
succeeds (no Errno 111), the Tailscale peer reads Online:true, and a keyed
POST /v1/chat/completions returns HTTP 200. Page re-verified (updated/verified_on ->
2026-11-06, review_after -> 2026-12-06) with the probe recorded on the page.
New trap recorded on the page and in systems/scheduled-tasks.md: on omniroute-1 the
RAW model id deepseek/deepseek-v4-flash returns HTTP 401 "No active credentials for
provider: deepseek", while the fleet's deepseek-paid-smart alias returns HTTP 200 and
resolves to that same model. The 401 is an alias-routing artefact, not an outage - probe
the alias the fleet pins, never the raw model id. (A first probe using the wrong
key_env also produced a 401; omni-101's key env is HERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
Freshness re-run after the edits: 0 stale / 124 fresh / 3 no-date of 127 pages.
Quartz rebuilt and redeployed. Vault mirror synced to both conventions (sha-verified).
ROOT CAUSE FIXED - systems/skills-index.md had TWO WRITERS:
the repo wiki-maintain.sh (302 skills / 68 categories) and the stale
~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26
categories - it used Path.rglob() and counted the parked .archive/ tree) were
different scripts with different generators, and wiki-maintenance/SKILL.md
line 569 pointed the 23:59 nightly audit at the stale path while the 03:00
stocktake ran the repo path. Each job overwrote the other's page; neither
raised an error. The stale copy is now a delegating wrapper with no generator
logic, and the repo script gained --no-build so the wrapper honours the old
interface without a second generator. Verified by execution.
Live-probe corrections:
- infrastructure/docker-services.md: CT200 61 running / 68 total (was 62);
worldmonitor-seeder-scheduled does not exist on ANY host (zero 'seeder'
matches across CT200/201/333/460); CT333 first container renamed again
(nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1,
CT1001 1/9.
- infrastructure/tailscale.md: node table verified programmatically - 52 rows
vs 52 live IPv4 peers, no live peer missing, no orphan row, every
online/offline state matching. conduit 59d->60d, ubuntu-server-xfce
75d->76d. Serve table still matches tailscale serve status exactly.
- infrastructure/hosts.md: skills figure corrected to 302 / 68 categories.
- index.md: skills-index bullet refreshed to the 2026-11-05 figures.
New incident: incidents/2026-11-05-skills-index-generator-divergence.md
(closes the still-open item on the 2026-08-15 skills-index drift incident).
Freshness: 0 stale / 123 fresh / 3 no-date of 126 pages.
BUILD FIX — the stocktake script was failing at step 3/3 with an opaque esbuild panic.
Root cause: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an
unquoted YAML title containing ": " (from "voice mode: local"), which YAML parsed as a
mapping entry and Quartz rejected with "bad indentation of a mapping entry (2:27)".
A scan of all 117 pages found exactly one such line; it is now properly quoted.
Quartz now builds cleanly: 690 files emitted and deployed.
RESOLVED — incidents/2026-10-09-memory-recall-relevance.md (was the only stale page).
The page claimed the recall-ranking fix "was not present in live config" and that
conflict 152 was "the only open conflict". Both claims were false:
- The weights ARE persisted, in ~/.hermes/mnemosyne/config.yaml (vec_weight 0.5,
fts_weight 0.3, importance_weight 0.3) plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 —
not in ~/.hermes/config.yaml's memory.mnemosyne block, which is what the
2026-10-10 probe read. The page was the reason the incident looked open for 3 weeks.
- The original regression test now passes: "mnemosyne recall 'Are you up to date'"
returns only substantive current-state facts (scores 0.174/0.130/0.123); the stale
Uber Eats episode no longer outranks anything.
- Conflict 152 was auto-resolved 2026-10-29 23:53:47Z as extraction noise.
All three open items closed; page moved to status: resolved.
LIVE DRIFT CORRECTED (all verified against live systems on 2026-11-01):
- infrastructure/tailscale.md: omarchymac went offline (genuine transition, ping
times out); m93p peer record reads false for the 4th consecutive time while the
host pongs and SSH returns up 1d12h, so the row stays online; 5 offline ages
refreshed; no IP changes, no missing peers.
- infrastructure/docker-services.md: CT201's Tailscale IP was wrong
(100.116.116.116 -> 100.96.244.39); 3 misspelled overchat-* entries and 1 stale
worldmonitor-seeder-fast entry removed from the CT200 list, which is now
regenerated verbatim from live docker ps; CT333's adoring_jemison renamed to
thirsty_payne.
- infrastructure/proxmox.md: storage percentages refreshed on both hosts from live
pvesm status; pct/qm lists re-confirmed (no CT601 anywhere); m93p uptime and an
elevated load average recorded.
- systems/mnemosyne.md, systems/omniroute.md, systems/messaging-integrations.md:
re-probed and verified_on bumped.
- systems/skills-index.md: regenerated — 301 unique skills in 68 categories,
46 archived. (301 is the unique (category, skill_name) count; find returns 302
files due to the known nested obsidian-git-sync alias duplicate.)
Freshness report: 0 stale / 114 fresh / 3 no-date of 117 pages.
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).
Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.
- infrastructure/tailscale: node table rebuilt from tailscale status --json.
50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
re-verification. m93p records online (peer record chronically stale, third
occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
Seven job rows carried stale error claims for jobs since cleared to ok --
each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
failure notice was never delivered (deliveries.db has zero rows for
2026-10-30), so the incident row is the only trace. Job not re-enabled and
gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
tab bar is built from the service groups (renderTabs iterates state.groups),
so the settings.yaml layout block and its tab:/columns: keys are inert --
there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
amber; manager.py runs inside the homepage-manager container; and the "no
Traefik container" evidence was wrong -- the container exists but is
stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
occupy 443 stands. Same evidence error corrected in tailscale.md.
Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).
No secrets written -- provider names, key names and HTTP status codes only.
Wiki maintenance (wiki-maintain.sh, exit 0) + live-vs-page drift audit.
Script:
- systems/skills-index.md regenerated from the live skill tree: 302 skills in 69
categories; 45 archived skills under ~/.hermes/skills/.archive/ now excluded and
noted. (303 SKILL.md files exist outside .archive/, but
productivity/obsidian-git-sync/obsidian-git-sync is a nested alias duplicate,
so 302 unique is correct.)
- Freshness report raw/stocktake/freshness-20261029.txt: 2 stale / 105 fresh /
3 no-date. After the corrections below and a second script run: 0 stale /
107 fresh / 3 no-date.
- Quartz rebuilt (110 pages -> 644 files) and deployed to public/.
Drift corrections (live-verified):
- infrastructure/tailscale.md: State column rebuilt from tailscale status --json,
11 rows changed. Two were state transitions, not age drift:
* sftpgo (CT299, pve) is back online - tailscale ping pong in 0s,
systemctl is-active sftpgo -> active, tailscale ip -4 inside the guest.
* m93p is online despite a stale peer record (Online: false in the JSON) -
tailscale ping pong in 1ms, SSH over the tailnet returned up 11 days.
Second occurrence of this pattern after 2026-10-26.
The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged
at 54 (53 live IPv4 peers + hermes); 76 peers total, 23 IPv6-only funnel
ingress. No missing peers, no IP mismatches. Container identity re-confirmed
from inside all 34 guests via pct exec tailscale ip -4 on both hypervisors;
CT601 still absent from both.
- incidents/2026-10-01-cliproxy-connection-auth-expired.md: the residual claim
"the connection credential itself was never re-authenticated" is falsified.
Live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128
(CT600) returns HTTP 200 - the connection has since been re-authenticated. The
same pin on custom:omni-101 (CT101) still returns HTTP 401, so the nodes carry
separate credentials. Claim marked superseded, not deleted (SCHEMA rule 9); the
remediation was never reversed and remains correct.
- incidents/2026-10-05-dead-model-pin-repin.md: every claim re-probed. All 7 jobs
still on custom:omni-101 with routing aliases; both aliases HTTP 200;
ollama-cloud/qwen3.5:397b still HTTP 410; nvidia/deepseek-v4-flash still
HTTP 400; command-code/gpt-5.6-terra now HTTP 401 rather than the original 403
- still dead, different failure signature.
- systems/scheduled-tasks.md: the Known-Issues entry repeating the
"never re-authenticated" claim now carries the correction. Fleet figures
re-checked against live jobs.json and remain accurate (29 jobs / 26 enabled /
3 disabled, 5 non-clean last_status).
- log.md / index.md: stocktake entry appended and index last-updated refreshed.
Verified correct, no change: systems/homepage-dashboard (CT201 has 8084 and 8085
listening, no 8082, Homarr on 8102); CT201 Serve routes (/ -> 8085, /mgr -> 8084
on both :443 and :8443, Steel routes still gone); CT460 Serve routes (12 paths
on :443, plus :8443/webhook Funnel and :9443).
Noted for a human, not performed: vault mirror at /home/hermes/wiki/agent-estate-wiki
has drifted on both conventions (top-level 35 match / 3 stale / 72 missing;
wiki/ 41 match / 53 stale / 16 missing). A full mirror pass is larger than a
stocktake's mandate.
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).
Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).
Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.
New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
Mark-don't-delete: the CT450 '3 commits behind' follow-up and the two
mnemosyne-vector-recall follow-ups are struck through with their verified
2026-10-26 resolution rather than removed.
- CT450 tracks origin/main (there is no origin/master on that remote) and is
0 commits behind.
- Coverage monitor b63ed5817212 is enabled, last_status ok, working coverage
96.8% — correctly has nothing to fire on.
- fastembed 0.8.1 now imports in both managed runtimes plus the legacy venv.
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted,
service restarted). The freshness report listed 8 stale pages; each was diffed
against live state and corrected only where reality could be verified. A second
script run rewrote the same-day report with 0 stale / 103 fresh / 4 no-date,
so the committed report matches the tree it ships with.
- infrastructure/hosts.md: CT600 is LIVE (not retired); CT601 does not exist on
either hypervisor; CT333 verified running; both hypervisors verified; config
33->51; skills 66->302; cron 25->28 (25 enabled / 3 disabled); Xvfb :102->:99;
providers split into Omniroute:20128 (CT600) and omni-101 (CT101, default).
- agents/hermes-production.md: Tailscale IP 100.96.100.82 -> 100.118.5.51 (the
old value is the PVE hypervisor); v0.21.5 -> v0.22.3; dead 8082 claim removed;
fallback chain rewritten from live config (first entry is the dead nous
credential); all 8 auxiliary slots now no-think/freeapi/auto; cron 30->28.
- agents/hermes-chaos.md: v0.22.3 (same build as CT460); "3 commits behind" ->
0 behind origin/master; default model smart-route on custom:omni-101; cron
"11 paused" -> 25 jobs (13 enabled / 12 disabled), 9 names shared with CT460.
- infrastructure/tailscale.md: node table rebuilt from status --json; octoagent
and omarchymac-1 were missing entirely (52 -> 54 non-funnel nodes); m93p now
online; every offline age refreshed; omni row corrected to "CT601 removed
from both hypervisors"; Serve tables re-probed per host.
- systems/memory-backend.md: re-verified against live config; coverage 1899/2350
(80.8%), working 96.8%, episodic 99.4% — above the 95% reindex threshold.
- runbooks/provider-health.md: the step-3 OmniRoute snippet raised IndexError
(provider renamed omniroute -> Omniroute:20128; key moved to key_env) — fixed;
new step 3b documents the omni-101 /v1/models false-failure quirk.
- runbooks/update-hermes-safely.md: added managed-runtime dependency re-assert
step; clarified --branch vs stable channel (v0.22.5 available).
- incidents/2026-10-03-ct450-hermes-update.md + incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md:
Current-state sections recording current versions and embedding coverage.
All 9 corrected pages verified 200 at http://127.0.0.1:9120/<path>.
Vault mirror at /home/hermes/wiki synced for all touched files (22 copies,
sha256-verified) and committed locally; the vault is local-only and is not pushed.
No secrets written — IDs, URLs, key env var names, counts and timestamps only.
- skills-index.md regenerated from live skill tree: 302 skills / 69 categories
(4 skills moved to .archive/ since 10-24), 45 archived
- freshness-20261025.txt: 0 stale, 101 fresh, 4 no-date of 105 pages
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json
(springroll-browser was missing; omarchybook + openmausbot now offline; all offline
ages refreshed; homeassistant exit-node wording corrected) + CT201 serve routes
corrected (443 AND 8443; /steel only on :443)
- systems/homepage-dashboard.md: same CT201 route + false Traefik claim corrected
- reports/communities-wa-social-media-audit-2026.md: re-verified against live WA source
- incidents/2026-08-26-work-outlook-basic-auth-disabled.md: re-probed, still open
- log.md: appended stocktake entry
Maintenance script ran clean (skills-index 346 -> 347, freshness audit, Quartz rebuilt
596 files and deployed, hermes-wiki-serve.service restarted).
infrastructure/tailscale.md — node table rebuilt from live `tailscale status --json`.
The 2026-10-19 table had drifted: octop (100.92.163.122) and springroll
(100.100.153.125) missing entirely, openmausbot / openmausbot-vm present in prose
but absent from the table body, omarchymac now offline, all offline ages stale.
Rebuilt keyed on DNS name rather than the JSON HostName — HostName is `ct260` for
the freshrss node and BOTH OmniRoute nodes report the hostname `omniroute`, which
collides silently. Now 51 non-funnel nodes. Container states cross-checked against
live `pct list` on pve and m93p: CT255 stopped, CT601 stopped, CT299 running.
reports/uber-eats/ — 2026-08 and 2026-09 recovered. The generator wrote only to
~/wiki/reports/uber-eats/ and never to the canonical tree, so both months were
missing from the published wiki. Root cause fixed in
~/.hermes/cron/scripts/generate-uber-eats-report.py: it now targets
~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree,
and now emits YAML frontmatter (without it the pages landed in the audit's
"no updated DATE" bucket and Quartz could not read the page date). 2026-08
previous-month column corrected from $0.00 to $264.51 / 11 orders.
incidents/2026-08-15-skills-index-drift.md — re-verified live; the named
regenerate-skills-index.sh no longer exists on disk (regeneration now lives in
wiki-maintain.sh step 1/3). Incident prevention confirmed working: 347 skills.
incidents/2026-09-02-sftpgo-outage.md — re-verified live: CT299 running, health
endpoint HTTP 401 (auth required, service up), no recurrence.
reports/communities-wa-social-media-audit-2026.md — left stale (54d). Its source
document no longer exists on disk, so the baseline cannot be re-verified against
a live source. Flagged, not edited.
Freshness after this run: 1 stale, 94 fresh, 6 no-date of 101 pages.
omniroute-1 (100.78.63.20:20128, provider custom:omni-101) TCP-unreachable 10-14 08:30 -> 10-15 18:07
(httpcore.ConnectError Errno 111 -> openai.APIConnectionError -> RuntimeError: Connection error).
36 failure rows across 13 jobs = 24 genuine model-call failures on 12 jobs + 12 Mnemosyne Watchdog
symptom rows. 10-14 160/12, 10-15 151/24 (vs 10-12 175/0, 10-13 167/1).
STRUCTURAL: all 18 enabled agent-mode jobs are pinned to custom:omni-101; the only other model-capable
pin (custom:omni, CT600) sits on a no_agent job -> no model-call failover, and zero model calls
completed in 34h. Corrects a first-pass reading of Evening Stand-Down's green rows as CT600 evidence.
New: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (resolved, verified live).
Updated: systems/scheduled-tasks (frontmatter, Count, new Re-verified block), systems/omniroute,
systems/mnemosyne (monitor blast-radius note), current-state, index, log.