Files
agent-estate-wiki/log.md
T
Tony0410 fda56e53fe audit 2026-11-02: Weekly Review lost to gateway handoff failure; three cron workers SIGTERM'd
Window 2026-11-01 00:00 -> 2026-11-02 00:4x AWST. 167 completed / 1 running / 2 unknown.

NEW incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (active)
NEW incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (resolved)
UPDATED systems/scheduled-tasks.md (Count -> 2026-11-02 window, Re-verified section)
UPDATED systems/messaging-integrations.md (Telegram flapping, relay adapter absent ~29h)
UPDATED index.md, log.md
2026-11-02 00:27:09 +08:00

196 KiB
Raw Blame History

Wiki Log

Chronological record of wiki actions. Append-only.

[2026-07-22] init | Fresh Hermes Systems Wiki

  • Created technical wiki at /home/hermes/wiki
  • Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
  • Wrote SCHEMA.md with 11 operating rules
  • Wrote index.md as curated Start here dashboard
  • Reseeded system/infrastructure/runbook stubs for existing homelab
  • Git initialized, local-only
  • No secrets, passwords, tokens, or private keys written to vault

[2026-07-22] audit | External audit pass (Claude, at Anthony's request)

  • Merged duplicate pages: systems/omniRoute.md (empty stub) -> kept systems/omniroute.md (populated); repointed 5 links
  • Merged duplicate pages: runbooks/safely-update-hermes.md -> kept as runbooks/update-hermes-safely.md (fuller content); repointed 2 links
  • Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
  • Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
  • Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
  • Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
  • Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts

[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)

  • Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
  • Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
  • Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
  • Fixed a broken review_after: 0001-01-01 in decisions/keep-openrouter-as-fallback.md
  • New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
  • queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate

[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)

  • Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
  • Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
  • Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
  • Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
  • Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
  • Confirmed: Mnemosyne is, and has only ever been, this box's memory backend

[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)

  • Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
  • These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki

[2026-07-25] operations | Tool Search and gateway resilience

  • Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
  • Documented how new skill categories are derived from the first folder below ~/.hermes/skills/
  • Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
  • Recorded the I/O-controller delegation limitation so an ineffective IOWeight is not mistaken for active protection
  • Added health checks, exact effective settings, backup location, and rollback procedure
  • No secrets, tokens, or private credentials were written to the wiki

[2026-07-28] dashboard | Jobs Dashboard feature additions

  • Fixed bulk status update (payload shape mismatch: {ids, status} → {ids, updates: {status}})
  • Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
  • Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
  • Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
  • Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion Notes / JD Summary
  • Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
  • Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
  • Updated Synergy and Dept of Communities to Phone Screen status
  • Updated wiki runbook with full dashboard feature documentation and API endpoint table

[2026-07-27] incident | Job Radar search and dashboard recovery

  • Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
  • Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require MISSING_COVER_LETTER_COUNT=0
  • Recorded the production cron model (gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5)
  • Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
  • Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both :443 Serve and :8443/webhook Funnel
  • Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
  • Verified no secrets, API keys or private credentials were written to the wiki

[2026-07-29] update | Steel Browser deployment — browser backend switch

  • New page: systems/browser-backend.md — rewritten to reflect Steel Browser as primary, Camoufox as fallback
  • New page: comparisons/steel-vs-camoufox.md — comparison of the two backends
  • Updated: infrastructure/hosts.md — added CT201 and CT460 to verified hosts table
  • Updated: infrastructure/docker-services.md — added Steel Browser compose file
  • Updated: infrastructure/tailscale.md — added CT201 Steel serve routes
  • Updated: index.md — browser-backend link description
  • Updated: current-state.md — added 2026-07-29 changes section
  • Config change: browser.cdp_url set to http://100.96.244.39:9223 in ops profile

[2026-07-30] update | Open WebUI updated on CT245

  • Updated image to ghcr.io/open-webui/open-webui:main (new digest e97bf9531916, Jul 27)
  • Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
  • Restarted container after OOM hang
  • Updated infrastructure/proxmox.md

[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed

  • Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
  • Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
  • Ran mnemosyne reindex --yes to rebuild all embeddings — working memory now at 100%
  • Updated health check script (mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py)
  • Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
  • Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
  • DB grew from 8.7M to 21M with the new embeddings

[2026-08-01] incident | Mnemosyne capture pipeline repaired

  • Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no working_memory row existed.
  • Corrected stringified structured YAML (sync_roles, skip_contexts, reflect, sleep_threshold, ignore_patterns) and enabled profile isolation; hermes config check passed.
  • Verified the repair with a second CLI marker that persisted as an exact normalized [USER] database row.
  • Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route :9119; added authoritative identity/preference/workflow slots.
  • Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
  • Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.

[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation

[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync

  • Ops profile (~/.hermes/profiles/ops/config.yaml): enabled mnemosyne plugin with profile_isolation: true, allow_tool_override: false, shared_surface_read: false
  • Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
  • Verified end-to-end: memory writes land correctly in both profiles independently
  • Updated systems/mnemosyne — profile isolation section now documents ops config
  • Updated current-state — added ops profile Mnemosyne entry
  • Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
  • Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision

[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation

  • Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances
  • Reworked deepseek-flash-free and good-free on CT600: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort
  • Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
  • Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
  • Updated systems/omniroute — added Recent Changes (2026-08-04) section
  • Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated updated date

[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite

  • OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. is_active=0 does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
  • Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
  • Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
  • Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
  • Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
  • Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
  • Updated index.md — last-updated date 2026-08-04 → 2026-08-07

[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review

  • Portainer password reset documented: CT245 admin user confirmed as anthony (not admin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state.
  • SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint, // root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.

[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review

  • Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker from freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named jobs (e.g. vertex/gemini-2.5-flash, deepseek-v4-flash-free) → smart-route. Daily Marketing Job Radar → gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run (ID a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, CT333 Nanobot has live jobs.json with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
  • Updated current-state — added 2026-08-08 changes section
  • Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues

[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue

  • Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
    • TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
    • RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
  • Root cause identified: Disk at 83% (11GB free on 65GB), state.db at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
  • Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
  • Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
  • OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
  • No new pages created — all changes applied to existing pages

[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals

  • OmniRoute AutoRoute variants documented: auto/best-free (Auto Cheap) and auto/coding-free are virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: auto/best-free routed to Antigravity. Updated systems/omniroute.
  • OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains memory.provider: mnemosyne. Added to comparisons/memory-backends.
  • FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default freellmapi-auto. Added to comparisons/model-routers.
  • No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.

[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration

  • OmniRoute migrated from CT600 to CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list, grep, Tailscale DNS).
  • Updated systems/omniroute — Current Configuration rewritten for omni/CT601, added Recent Changes (2026-08-13), bumped updated/verified_on to 2026-08-13.
  • Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped updated.
  • Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
  • Updated infrastructure/hosts — OmniRoute proxy entry → omni; CT600 reference marked verified/retired.
  • Updated index — last-updated date 2026-08-10 → 2026-08-13.
  • No new pages created; all changes applied to existing pages. No secrets written.

[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated

  • Skills index drift incident: systems/skills-index.md claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating regenerate-skills-index.sh script, and adding cron job 1a8e90ea2271 (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift.
  • Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped updated/verified_on to 2026-08-15.
  • Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
  • Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
  • Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
  • Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
  • No secrets written.

[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash

  • New page: systems/nextcloud.md — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
  • Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
  • Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
  • Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
  • No secrets written.

[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk

  • OmniRoute upstream capacity pressure documented: ServiceUnavailableError: Structurally heavy chat request capacity is busy reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
  • Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
  • Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config http://omni:20128/v1 is correct — /v1 is required.
  • Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped updated/verified_on to 2026-08-16.
  • Updated current-state — added Recent Changes (2026-08-16) section, bumped updated to 2026-08-16.
  • Updated index.md — bumped last-updated date to 2026-08-16.
  • No new pages created. No secrets written.

[2026-08-18] incident | Mnemosyne capture pipeline broken (active)

  • New page: incidents/2026-08-18-mnemosyne-capture-pipeline — working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
  • Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped updated/verified_on to 2026-08-18, added cross-reference to Related.
  • Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped updated to 2026-08-18.
  • Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
  • Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from active to resolved (that earlier incident was fixed).
  • No secrets written.

[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history

  • Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
    • wiki-stocktake-maintenance (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from cx/gpt-5.6-luna → go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.
    • CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
    • Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
  • Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped updated/verified_on to 2026-08-23.
  • Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped updated/verified_on to 2026-08-23.
  • Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
  • Updated index.md — bumped last-updated date to 2026-08-23.
  • No secrets written.

[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status

  • New page: systems/ignis.md — documented Ignis browser-native Obsidian (Docker ignis-subpath:0.8.9, port 8080, Tailscale Serve at hermes.kangaroo-eel.ts.net/ignis). Volume mount: /home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active.
  • Updated systems/model-providers — added command-code provider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumped updated/verified_on to 2026-08-19.
  • Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider, deepseek-flash-mix combo, and Tavili search proxy status. Bumped updated/verified_on to 2026-08-19.
  • Updated infrastructure/docker-services — added Ignis compose file and service definition.
  • Updated infrastructure/hosts — added Ignis to CT460 row.
  • Updated systems/obsidian — added Ignis as browser interface, corrected vault path to agent-estate-wiki/, added Quartz static site reference.
  • Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
  • Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
  • No secrets written.

[2026-08-26] update | Nightly wiki audit — Cron job reconciliation

  • Updated systems/scheduled-tasks — reconciled live hermes cron list (27 jobs) against wiki table (29 entries). Removed: Workday start reminder (no longer in live list), CT 221 Miner Watchdog (job d5516d0a5147 removed; CT221 status unknown), Review Agent Estate Phase One (one-time job completed 2026-08-08). Added: weekly-save-restricted-bot-restart (0 3 * * 0, local, no-agent). Updated count from 29 to 27.
  • Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
  • Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
  • Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
  • No secrets written.

[2026-09-02] incident | SFTPGo outage on CT299

  • New page: incidents/2026-09-02-sftpgo-outage.md — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), tailscaled died; PVE pct start failed due to 5 stale lxc-attach processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop 299 --skiplock + pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
  • Updated current-state — added Recent Changes (2026-09-02) section, bumped updated/verified_on to 2026-09-04.
  • Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
  • No secrets written.

[2026-08-28] update | Department of Communities role and social-media audit

  • New page: reports/communities-wa-social-media-audit-2026 — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
  • Updated systems/current-profile — recorded current employment context and linked the report.
  • Updated index.md — added the report to the Reports section and refreshed the last-updated date.
  • No secrets written.

[2026-09-20] update | Nightly wiki audit — No changes in past 24h

  • No new pages created — no new systems, decisions, or incidents in the past 24 hours.
  • No existing pages updated — all wiki content remains current.
  • Freshness audit run: freshness-20260920.txt generated at 03:01 AWST (scheduled cron).
  • Skills index auto-regenerated: systems/skills-index.md updated at 03:01 AWST (scheduled cron 1a8e90ea2271).
  • Scheduled tasks reconciled: systems/scheduled-tasks.md updated at 03:02 AWST — 26 active jobs verified from live hermes cron list.
  • No secrets written.

[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation

  • New page incidents/2026-09-28-hermes-managed-runtime-deps — the 2026-09-28 → 30 cron-fleet outage: jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused by ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in the uv-managed runtime venvs under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (all 10 venvs, pip install --target), run every 15 min by job 96eb95ce12f2.
  • Updated systems/scheduled-tasks.md — table rebuilt from live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to d40085da631c, f77b7e46bd09, 0a7f933d06e2. Known Issues refreshed: job-radar 403 MODEL_NOT_IN_PLAN (streak 5), Telegram thread-delivery failures, weekly-save-restricted-bot-restart script restored, residual pre-fix dispatch errors on cbfe434a3b5e/1a8e90ea2271.
  • Updated systems/hermes-agent.md — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model custom → http://100.78.63.20:20128 / smart-route; skills 330 → 341; cron count 29 → 30.
  • Updated current-state.md — _config_version: 46 and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
  • Updated runbooks/job-radar.md — corrected primary model row (custom:omni-101 / gpt-5.6-terra via command-code) and added the 403 MODEL_NOT_IN_PLAN failure mode with recovery steps.
  • Updated systems/messaging-integrations.md — Telegram allowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
  • Updated index.md — last-updated to 2026-09-30, cron count to 30, new incident listed.
  • No secrets written.

[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene

  • No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
  • Updated systems/scheduled-tasks.md — table and counts re-verified against live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — dca8482e4f76 (403 MODEL_NOT_IN_PLAN, ×5), 1a8e90ea2271 (HTTP 410, model retired upstream, ×7), fad5a86979d4 (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: 1b23c042e17d and c8d52904e1d6 delivery targets now telegram:-1004321904721:102; c0a9fa314542 flagged for a revoked Telegram bot token; bare-telegram targets noted on 0e753e9720fd, fad5a86979d4, 066bb5191280. Known Issues restructured into Resolved / Open / Historical.
  • Updated systems/messaging-integrations.md — new Delivery target semantics section: a bare deliver: "telegram" resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails with Message thread not found and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
  • Updated systems/model-providers.md — provider stack rebuilt from live config.yaml: 5 custom providers (omni-101 5,384 models, Omniroute:20128 5,398, litellm 1,728, command-code 71, guacano 46); custom:omni recorded as removed. Default model corrected resilient-free → smart-route. New Retired / Unavailable Models table (retired qwen3.5:397b, plan-gated gpt-5.6-terra, catalog-missing deepseek-v4-flash) with the lesson that a raw model pin never self-recovers.
  • Updated systems/mnemosyne.md — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as [USER] [Anthony Martin] memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
  • Updated systems/hermes-agent.md — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
  • Updated index.md and current-state.md — last-updated bumped to 2026-10-01 with a Recent Changes block.
  • Vault sync: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under wiki/, not at the tree root, and index.md sits at the vault root.
  • No secrets written.

[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split

  • New page: incidents/2026-10-01-cliproxy-connection-auth-expired.md — job 06a0a1c8704f (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard (provider custom:Omniroute:20128, model cliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d, 79f9bb41ecf5).
  • Updated systems/scheduled-tasks.md — counts re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01: 06a0a1c8704f (HTTP 401), 0e753e9720fd (HTTP 400 — same NVIDIA model pin as the already-failing fad5a86979d4, confirming the shared-pin diagnosis), 066bb5191280 (delivery failed; report itself generated). Streaks advanced: dca8482e4f76 ×5→×6, 1a8e90ea2271 ×7→×8, fad5a86979d4 ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
  • Updated systems/messaging-integrations.md — corrected a wrong page fact. The page previously stated that a bad thread_id always loses the message. The 2026-10-01 records show two variants: Message thread not found (hard failure, message lost — the 2026-09-30 case) and configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id (a thread-less fallback send does occur, yet the run is still recorded delivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegram jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
  • Updated systems/current-profile.md — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no ~/.hermes/profiles/default/ directory (the default profile is the top-level config; ops is the only named sub-profile), the model default is smart-route (not freellmapi-auto), the default provider is custom → http://100.78.63.20:20128 / OmniRoute (litellm is now only the last fallback entry), and the config is _config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
  • Updated current-state.md — new Recent Changes (2026-10-02) block; frontmatter dates bumped.
  • Updated index.md — last-updated to 2026-10-02; cron line now notes the failing count.
  • Structural finding — the wiki exists as two trees and they have drifted both ways. /home/hermes/workspace/agent-estate-wiki is canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads (pages under wiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions — systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md and reports/communities-wa-social-media-audit-2026.md exist only in canonical — so canonical index.md currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), failing set still 6 but the composition moved. 1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard that the cliprox incident predicted — making it 2 of the 3 cliprox/terra jobs confirmed broken, which retires any "one-job fluke" reading. 79f9bb41ecf5 is the last untested job on that pin (next due 2026-10-04 18:00). Streak recount against executions.db: dca8482e4f76 ×6→×7, but 1a8e90ea2271 ×8→×7 — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). Resolved: 1b23c042e17d's stale delivery_failed flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). Downgraded: the GitHub Trending Radar revoked-token claim is now "unverified at last run" — c0a9fa314542's 2026-09-30 19:22 run completed and delivered. New Known Issue recording that a delivered notice is not a successful job (both cliprox failures read delivery_outcome: delivered on a failed status).
  • New page incidents/2026-08-26-work-outlook-basic-auth-disabled.md — the work Outlook mailbox has been unreachable since 2026-08-26: himalaya is configured for password auth while the tenant advertises only XOAuth2, so himalaya -a work dies with login not supported. TCP/TLS to outlook.office365.com:993 is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
  • Updated systems/messaging-integrations.md — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, work unreachable; frontmatter updated/verified_on bumped. Added the tooling consequence: email/mail-lookup resolves the account before searching, so work-mail requests hit this wall and must be reported as account unreachable, not "no results found".
  • Updated systems/hermes-agent.md — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to six jobs across three model faults plus one credential fault, and a second entry added for the stock-take job being down; the Email: connected bullet corrected.
  • Updated current-state.md and index.md — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
  • systems/skills-index.md regenerated 341 → 344 — including a new skill, fleet-wide-failure-triage (devops). This was done by hand: job 1a8e90ea2271 is still blocked on HTTP 410 (retired qwen3.5:397b) and cannot self-recover. wiki-maintain.sh itself is healthy and was run manually, also writing raw/stocktake/freshness-20261003.txt — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under wiki/; index.md sits at the vault root.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair

Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.

New pages

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md — mnemosyne_recall had been scoring keyword-only: fastembed was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 without raising an error. Repaired 2026-10-03; the runtime guard now asserts fastembed + FASTEMBED_CACHE_PATH + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
  • incidents/2026-10-03-ct450-hermes-update.md — CT450 (hermesagent) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (ssh root@100.96.100.82 → pct exec 450 -- …) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).

Updated

  • current-state.md — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the mnemosyne doctor / sqlite_vec misread gotcha.
  • systems/mnemosyne.md — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job b63ed5817212 documented.
  • systems/hermes-agent.md — version corrected to v0.21.5+6169.gab86a20 (was v0.21.5+3837.gbfda74c), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no profiles/default/; ops is the only named sub-profile), Known Issues bullet added.
  • systems/memory-backend.md — Known Issues bullet for the FTS-only degradation.
  • systems/scheduled-tasks.md — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), 79f9bb41ecf5 due 2026-10-04 18:00; bare-telegram delivery warnings confirmed live.
  • agents/hermes-production.md — version, Python, profile, provider routing, cron count and messaging refreshed; stale custom:litellm / freellmapi-auto references replaced.
  • agents/hermes-chaos.md — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
  • infrastructure/hosts.md — CT450 row updated to v0.21.5.
  • runbooks/update-hermes-safely.md — new "Second instance — CT450" section (access path + pre-flight checks).
  • runbooks/provider-health.md — stale v0.19.0 version reference corrected.
  • index.md — last-updated bumped; both new incidents linked; cron-fleet line re-dated.

Note — mnemosyne doctor on this host prints vec0: present_but_unloadable and marks the vector tiers unavailable. That is a Python-3.11-legacy-venv artefact, not the gateway's state. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py.

[2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7

Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json + hermes cron list: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 6 → 7. The one new failure event in the window is 79f9bb41ecf5 (Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. It was the last untested job on the cliprox/terra pin, so that pin is now 3 of 3 confirmed broken — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271) and the Known Issues Open list all updated; frontmatter dates bumped.
  • Updated incidents/2026-10-01-cliproxy-connection-auth-expired.md — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still Open — needs the OmniRoute connection re-authenticated before any repin.
  • Updated current-state.md — new Recent Changes (2026-10-05) block: the cliprox 3/3 settlement, 1a8e90ea2271's repeat 410 on 2026-10-04, confirmation that no interactive sessions occurred in the window (all 10 active sessions were cron), and a streak-counting caveat.
  • Updated index.md — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
  • Streak-number caveat recorded: hermes cron list's "N failures in a row" is the failure_streak counter in jobs.json, not a DB count. ~/.hermes/cron/executions.db now retains only rows from 2026-09-30 onward (1,001 rows total), so executions.db recounts undercount older streaks — the earlier DB-based "×7" for 1a8e90ea2271 undercounts against the live ×11. Quote the live counter and note the retention floor.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source — wiki/systems/scheduled-tasks.md, wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md, current-state.md, index.md, log.md.
  • No secrets written; all credential-shaped values remain referenced by connection ID only.

[2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0

Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.

New

  • incidents/2026-10-05-dead-model-pin-repin.md — the seven-job dead-model-pin repin. Records the four dead pins (cliprox/terra HTTP 401 ×3, ollama-cloud/qwen3.5:397b HTTP 410 ×1, nvidia/deepseek-ai/deepseek-v4-flash HTTP 400 ×2, command-code/gpt-5.6-terra HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (free-stack ×6, smart-route ×1 on custom:omni-101) over another raw model name.

Updated

  • systems/scheduled-tasks.md — Count block rewritten: 0 failing (was 7), with all 30 failure_streak counters at 0; four jobs carry a stale pre-repin last_status: error that clears on their next scheduled run. Seven job-table rows updated (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd, dca8482e4f76), plus c0a9fa314542. New ### Resolved 2026-10-05 subsection in Known Issues; five resolved bullets removed from ### Open, leaving bare-telegram targets, the two-wiki-tree drift, the work-mailbox block, the stale custom:omni key and the delivered-notice rule as the standing open set.
  • index.md — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.
  • current-state.md — new Recent Changes (2026-10-06) block; frontmatter dates bumped.

Findings worth the ink

  • dca8482e4f76 (Daily Marketing Job Radar) delivered to topic 1030 at 23:56 — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for 79f9bb41ecf5 (23:39) and 06a0a1c8704f (23:45).
  • GitHub Trending Radar's revoked-token diagnosis is dead. c0a9fa314542 ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No getMe → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright.
  • delivery_outcome: delivered still does not mean the job worked. Every pre-repin failure recorded delivered, because the failure notice is itself the deliverable. Judge on the execution status column.
  • A stale notice opened the review. Anthony was prompted by a 85edf6772bca notice that had sat in topic 346 since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.
  • Window tally (executions.db, 2026-10-05 00:00 → 2026-10-06 00:15): 185 completed, 3 failed (all pre-repin, 03:00–07:00), 2 unknown (aborted first attempts of 79f9bb41ecf5 and dca8482e4f76 at 23:33/23:48, superseded by their successful re-runs) and 1 running (this audit).

Vault mirror: the new incident page plus systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day

Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — Count block corrected: the stale pre-repin last_status: error set is now three (1b23c042e17d, fad5a86979d4, 0e753e9720fd), not four — 1a8e90ea2271 cleared its own when it ran at 03:00. Added a Re-verified 2026-10-07 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 0 failed / 1 running; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
  • index.md — last-updated → 2026-10-07; cron-fleet line re-dated (0 failing as of 2026-10-07, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
  • current-state.md — new Recent Changes (2026-10-07) block; frontmatter dates bumped.

Findings worth the ink

  • 1a8e90ea2271 ran itself at 03:00 and proved the repin. wiki-stocktake-maintenance completed 2026-10-06 03:00:31 → 03:03:32 on custom:omni-101 / free-stack — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired qwen3.5:397b). It regenerated systems/skills-index.md (344 → 346 skills), wrote raw/stocktake/freshness-20261006.txt and committed to Gitea as 33f4e95. Its four HTTP-410 incidents (6df957a05ec6, d284168d3663, cda67de5b3e3, e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. The daily manual wiki-maintain.sh run can stop.
  • Vault mirror drift found and partly repaired. systems/skills-index.md in /home/hermes/wiki/agent-estate-wiki/ was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
  • Window tally (executions.db, 2026-10-06 00:00 → 2026-10-07 00:00): 176 completed, 0 failed, 1 running (this audit) — 177 rows. 14 sessions, all cron; no interactive (non-cron) sessions, and no new incident opened.

Vault mirror: systems/scheduled-tasks.md, systems/skills-index.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings

Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — new Re-verified 2026-10-08 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 1 failed / 1 running; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New Resolved 2026-10-07 Known Issues section, plus two new Open items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.
  • index.md — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.
  • current-state.md — new Recent Changes (2026-10-08) block; frontmatter dates bumped.

Findings worth the ink

  • The incident machinery had its first real workout since 2026-09-30 — and it worked. 62619aa6bcb3 (DHL Tracking) failed its 08:00 run inside ensure_browser() with TimeoutError: timed out on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident 62619a_f485ccfd4a0d (failure_type: timeout) was opened 08:00:52, alerted 08:00:58 and auto-resolved 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed.
  • The tracked parcel has arrived. The 14:00 run reported waybill 8052838382 as Delivered, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed.
  • Stocktake ran automatically for the second consecutive day (1a8e90ea2271, 2026-10-07 03:00, commit 8c63ef2; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once.
  • Two real defects surfaced from the stocktake's own log. (1) Step 3 of wiki-maintain.sh — the Quartz rebuild — is skipped on every run: there is no quartz.config.ts and no npx, so public/ is never refreshed and the published site silently ages. (2) skills-index under-reports by one: 347 SKILL.md files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md (the other duplicated basename, hermes-api-vault, survives because its two copies sit in different categories). Both are now Open items in systems/scheduled-tasks.
  • Unchanged, still carrying a stale last_status: error: 1b23c042e17d, fad5a86979d4, 0e753e9720fd — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-telegram targets and the custom:omni stale-key references are also unchanged.

Vault mirror: systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

[2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open

Window audited: 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days.

Pages written

  • incidents/2026-10-09-cron-provider-timeout-storm.md — new.
  • incidents/2026-10-09-memory-recall-relevance.md — new.
  • systems/scheduled-tasks.md — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
  • systems/mnemosyne.md — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
  • current-state.md — Recent Changes (2026-10-09) block; frontmatter dates bumped.
  • index.md — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
  • systems/skills-index.md + raw/stocktake/freshness-20261008.txt — the stranded 2026-10-08 03:01 stocktake artifacts committed.

Findings worth the ink

  • The fleet had its worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight remain open. The watchdog (96eb95ce12f2) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, i.e. it reported the storm rather than suffering it.
  • The 2026-10-08 window was never audited — this job's own 23:59 run failed with the same timeout (incident cbfe43_fa81052baaf7) and committed nothing.
  • The stocktake has a two-day gap. 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no freshness-20261009.txt exists. 1a8e90ea2271 is the one job still at failure_streak: 2.
  • An instruction was reported done and was not. Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". It is still enabled (paused_at: null) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
  • Memory recall relevance — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is not in live config (memory.mnemosyne carries no recall-weight keys; no top-level mnemosyne: key).

Vault mirror: the touched pages mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

2026-10-10 — Stocktake (manual run, wiki-maintain.sh)

  • systems/skills-index.md regenerated — 346 skills / 70 categories (unchanged count; frontmatter updated/verified_on refreshed to 2026-10-10). Live on-disk count is 347 SKILL.md; the one-skill gap is the known de-dup of productivity/obsidian-git-sync/obsidian-git-sync/.
  • Freshness audit → raw/stocktake/freshness-20261010.txt: 62 stale (≥21d), 23 fresh, 7 no-date of 92 pages. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No freshness-20261009.txt exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands.
  • current-state.md — new 2026-10-10 section. The 2026-10-09 timeout storm window re-tallied from executions.db (2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163 completed, 15 failed — the prior audit recorded 162/15+1 running; the stray running row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).
  • DHL monitor re-probed, still enabled. 62619aa6bcb3 — enabled: True, paused_at: null, 0 8,11,14,17,20 * * *, last_status: ok. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. No cron mutation performed by this audit — retirement stays open item #1.
  • infrastructure/tailscale.md verified against tailscale serve status — all 13 tailnet Serve routes plus the :8443/webhook Funnel match the page. No drift; not edited.
  • Not verified / left alone: the Quartz rebuild step (no quartz.config.ts, npx cannot fetch quartz — public/ was NOT refreshed, confirming the carried open item); docker ps (permission denied on /var/run/docker.sock from this session, so infrastructure/docker-services.md could not be probed and is left stale); the 80d 2026-07-22 cluster (no live source to check against).
  • Sessions in the 2026-10-09 window (state.db, epoch REAL): 19 total — 6 cron, 11 ios, 1 talaria-ios, 1 telegram. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the delivered count includes failure notices).

No secrets written; job/incident IDs only.

[2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-11T00:19). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 174 completed / 0 failed. Non-clean last_status 10 -> 5. Incidents: 7 closed during 10-10, 5 open (3 jobs). Open item #1 closed: the DHL monitor 62619aa6bcb3 is retired and verified absent from jobs.json by ID lookup (the 10-07 and 10-09 "deleted" claims were false).

Runtime/memory. The 2026-09-28 managed-runtime-dependency class recurred in the gateway env: mnemosyne (+native deps) and elevenlabs were missing from the committed uv env 795020dbf24343cd9c52df24723dca83 that activate_dependencies() loads. Repaired 21:26-21:34 and verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.

New page. systems/paseo - Paseo daemon v0.11.2 on 0.0.0.0:6767 (user unit paseo-daemon.service); recorded the daemon run vs daemon start supervisor trap, the /usr/bin/paseo 0.7.2 vs daemon 0.11.2 version split, and the truncated-cline false provider fault.

Platform. Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 answers neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.

Pages touched: systems/paseo.md (new), systems/scheduled-tasks.md, systems/hermes-agent.md, systems/mnemosyne.md, systems/messaging-integrations.md, incidents/2026-09-28-hermes-managed-runtime-deps.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-11] stocktake | Daily maintenance + live drift corrections

  • Ran wiki-maintain.sh: skills-index regenerated (346 skills / 70 categories), freshness audit written to raw/stocktake/freshness-20261011.txt (62 stale / 24 fresh / 7 no-date of 93 pages)
  • OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed. Verified via pct list, pct config, ss and tailscale status: CT600 omniroute (100.88.81.19) is RUNNING and serving :20128; CT601 omni (100.93.204.4) is STOPPED with onboot=0. Hermes resolves custom:omni → omniroute:20128 → CT600, not CT601. systems/omniroute.md rewritten with the live node table.
  • Third OmniRoute node found: omniroute-1 at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider omni-101 (5384 models, default smart-route, carries skills_hub). Not in pct list — identity unconfirmed.
  • Ignis duplicate found: a second ignis-ignis-1 container runs on CT200 (image ignis-server:latest, host port 8092, vault /opt/ignis/vaults), separate from CT460's ignis (port 8080, vault /home/hermes/wiki/agent-estate-wiki). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them.
  • Tailscale route table corrected against live tailscale serve status: /agent-estate target is bare http://127.0.0.1:8769 (not .../8769/agent-estate-role-dashboard.html); previously-missing tailnet-only :9443 → 127.0.0.1:5173 documented. CT201 routes refreshed (8085 targets + /steel 5173 + /steel-api 3000).
  • Proxmox inventory rebuilt from live pct list + qm list: added CT475 agentzero and VM110 omarchy; corrected 282 ubuntu-server-xfce to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). VM403 open-claw and VM401 hermes-old are absent from qm list — flagged unconfirmed rather than guessed.
  • Nanobot verified on CT333: gateway nanobot-gateway (PID 2259048) listening on 8765/8766; NOT a systemd nanobot.service/gateway.service (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified.
  • Docker services page corrected: the old "Docker runtime not available" note was a CT460 socket-permission artifact (/var/run/docker.sock is root:docker srw-rw----, hermes user not in group docker, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo).
  • Flagged for follow-up, root cause not investigated: mycompose-db-1 on CT200 is in Restarting (1) — crash-looping.
  • Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
  • Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.

[2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-12T00:10). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 175 completed / 0 failed (finalised), 10-11 171 completed / 0 failed + 1 running — two consecutive clean days. Non-clean last_status steady at 5; 0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected).

Corrected a false alarm instead of logging a fault. The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a naive-UTC vs AWST comparison error — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in systems/mnemosyne and systems/scheduled-tasks so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.

New content. queries/retained-research-answers was a 2026-07-22 stub and is now a real page — the 2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; -CompressLevel 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage was undocumented; the live datastore table plus host mount points are now in infrastructure/proxmox (/mnt/ssd0 93 % / ~63 GiB free is the pressure point).

Stale labels fixed. incidents/2026-08-18-mnemosyne-capture-pipeline carried status: active from August despite capture being verifiably healthy — now superseded, with the live evidence and a caution about the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open, and the skills-index blurb no longer claims no regeneration since 2026-10-07.

Verified, not changed. The blocked-page-recovery lesson that session reported patching (recovered: false = archive routes failed, not task failed; API-first rung) is present in the live skill.

Pages touched: systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/proxmox.md, queries/retained-research-answers.md, incidents/2026-08-18-mnemosyne-capture-pipeline.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections

Script run. wiki-maintain.sh completed all three stages: skills-index regenerated from the live skill tree (346 skills across 70 categories, unchanged count), freshness audit written to raw/stocktake/freshness-20261012.txt (54 stale / 32 fresh / 7 no-date of 93 pages), and the Quartz step took its documented no-quartz.config.ts path (build skipped, public/ unchanged — expected, not a failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and written its artifacts, so results were read from the report rather than assumed.

Drift corrected from live probes (2 pages):

  1. systems/nanobot — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on CT333 is back up, HTTP 200 on http://100.122.189.81:8766/. Port 8900 — the endpoint the disabled OmniRoute nanobot provider connection uses — is still closed, so status stays degraded rather than flipping to active. SSH to CT333 is unreachable from CT460, so whatsapp-bridge.service was not re-verified.
  2. systems/home-assistant — carried confidence: low plus a page-wide "cannot confirm HA runs" caveat from 2026-07-22. Live: HTTP 200 on http://100.68.139.5:8123/, Tailscale node homeassistant active and offering an exit node, VMID 100 running per qm list. Confidence raised low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).

Verified and left stale. systems/openrouter and systems/nextcloud were probed shallowly and nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.

Pages touched: systems/nanobot.md, systems/home-assistant.md, systems/skills-index.md, raw/stocktake/freshness-20261012.txt, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes

Window. 2026-10-12 00:00 → 23:59 AWST (sessions from ~/.hermes/state.db, cron from ~/.hermes/cron/executions.db + cron_incidents; live probes 2026-10-13 00:05–00:45). One former compaction of this job left part of the review written but the log entry and cross-page updates unfinished — completed here.

Headline 1 — a clean run hid a dead credential. c0a9fa314542 (GitHub Trending Radar — Weekly Roundup, Mondays 09:00) reported completed with delivery_outcome: delivered on 2026-10-12, but its fetch-then-send script died at digest.py send(). Live probe of the bot token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 from getMe — the token is a valid 46-character shape, i.e. revoked, not malformed. The fetch path is healthy (100 repos, 26 cache files in 24 h). Critically, this supersedes the 2026-10-06 entry on current-state, which concluded the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in systems/scheduled-tasks: a completed run does not clear a credential fault unless the run exercised the same call that failed, and green cron status is not delivery proof. (Credential was present; value neither recorded nor repeated.)

Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new). The 2026-10-12 07:01 Daily Marketing Job Radar run drove camofox-browser.service to 2.5G, swap fully consumed, available: 0B, 383 tasks against the drop-in ceiling MemoryHigh=2G / MemoryMax=3G / MemorySwapMax=1G. REST /tabs and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at 07:23:50, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis: workload exceeded the cgroup, not a leak — OOMPolicy=kill never fired.

Headline 3 — systems/browser-backend corrected against live probes. Prior revisions were wrong on three counts: the REST API listens on :9377 only (:9093 closed, :9378 does not exist — REST and WebSocket are the same service, not two), Camofox runs as a systemd --user unit on the Hermes host (CT460), not CT450, and the containment drop-in was undocumented. current-state.md also still described the old port model; both pages now match the probe.

Cron state. 2026-10-12: 174 completed / 0 failed / 1 running (this audit, cbfe434a3b5e, started 23:59:41) — third consecutive clean day; 2026-10-11 finalised at 172 / 0. Retention floor moved to 2026-10-07 08:15. Fleet 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3 (0e753e9720fd error, 1b23c042e17d error, 066bb5191280 delivery_failed). fad5a86979d4 (Facebook Birthday Weekly Registry Sync) and dca8482e4f76 (Daily Marketing Job Radar) both cleared on 2026-10-12 — the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.

Incidents — open 5 → 2. Closed on 2026-10-12: fad5a8_a13bbc837e1c + fad5a8_c80b59a907f0 at 03:05:39 (the two deepseek-v4-flash HTTP-400 rows, cleared by the registry sync's clean run) and dca848_23fcd2b18aaa at 07:33:41. Remaining: 1b23c0_c35703f637b8 (alerted — weekly job, cannot clear before Fri 2026-10-16 08:00) and 0e753e_e7f8194a7d06 (detected since 2026-10-01 — the job has no retained executions, so it cannot clear until 2026-11-01). → incidents/2026-10-09-cron-provider-timeout-storm updated from "8 still open" to 11 of 12 closed with the full closure timeline; the pattern recorded there is that each incident clears only on the next successful run of its own job, so weekly jobs carry their rows for up to a week after the fault is gone.

Mnemosyne (2026-10-12 12:02 health run, green). Working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 % — both above the 95 % floor, no reindex; orphan embeddings 0; FTS5 exactly aligned; 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → systems/mnemosyne.

Sessions. 18 in the window, all cron, zero interactive. The 03:00 stocktake 1a8e90ea2271 ran clean and pushed 0eee5f1 (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 categories — the known 347-vs-346 on-disk off-by-one persists).

Pages touched: current-state.md (new Recent Changes section + superseded-correction tag), systems/scheduled-tasks.md (frontmatter, job inventory, non-clean statuses, re-verified history, five job rows, ### Resolved 2026-10-12, corrected incident IDs, ### Open additions, new ## Re-verified 2026-10-13 block), systems/browser-backend.md (port + host + containment rewrite, history entries), systems/mnemosyne.md (12:02 coverage), incidents/2026-10-09-cron-provider-timeout-storm.md (status + full closure table + timeline), index.md (date, four blurbs), log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential

Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.

Cron. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight → the day finalises at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 — the first failed run since 2026-10-09 (which recorded 15). Retention floor moved to 2026-10-08 07:11 (MAX_TERMINAL_EXECUTIONS = 1000; 1,001 rows = 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads 126 / 1, partial).

Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent, alerted_at: NULL — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three on the opencode path (10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode; 10-09 timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00 run. → incidents/2026-10-05-dead-model-pin-repin

Finding 2 — the failure stranded its own generated artifacts (now committed). Step 1 (skills-index) and step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working tree: systems/skills-index.md (modified, date-only: updated/verified_on 10-12 → 10-13; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked; 48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit. Consequence: the stocktake published nothing on 10-13 — and because the agent row is what commits, even a perfect script run leaves nothing behind when the LLM turn dies.

Delivery. 159 suppressed / 9 delivered / 1 no-record. Delivered: Daily Marketing Job Radar, Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday, Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 2026-10-01.

Mnemosyne. Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no reindex was needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than bare; the delivery layer still suppressed it (delivery_outcome: suppressed) — the matcher tolerates wrapping rather than requiring the literal token as a strict control. → systems/mnemosyne

Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive session.

Fleet. 29 jobs (26 enabled / 3 disabled), pins unchanged; non-clean last_status 3 → 4 (cleared fad5a86979d4 10-12 03:05 and dca8482e4f76 10-13 07:02; added 1a8e90ea2271); open incidents 2 → 3 (1 alerted, 2 detected, 8 closed, 66 resolved; 77 rows).

Pages touched: systems/scheduled-tasks.md (frontmatter, retention floor, non-clean list, ## Re-verified 2026-10-14), systems/skills-index.md (committed stocktake artifact), current-state.md (new Recent Changes + date), index.md (date + two blurbs), log.md.

Vault mirror. The 2026-10-13 audit was never mirrored to /home/hermes/wiki (last mirror bb09581, 2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus skills-index.md, whose vault copy was still the 2026-10-08 version. The vault's own raw/stocktake/ still holds only the 2026-10-03 report.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt

Ran wiki-maintain.sh (03:01:09 → 03:01:10 AWST): step 1 regenerated systems/skills-index.md (346 skills / 70 categories), step 2 wrote raw/stocktake/freshness-20261014.txt (48 stale / 38 fresh / 7 no-date of 93 pages), step 3 skipped the Quartz rebuild again (no quartz.config.ts, no npx) so public/ is untouched — the known carried item.

Finding 1 — a second, unclustered Proxmox hypervisor exists: m93p (100.104.51.98), PVE 9.2.0 (kernel 7.0.14-12-pve). pvecm nodes is empty on both pve and m93p — no cluster — and guest IDs collide between them (m93p CT221 is gitea; pve CT103 is cliproxyapi; CT100 and CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the hypervisor port :8006 returned only m93p and pve. m93p had been sitting in the Tailscale node list for weeks as an unremarkable online peer, with no wiki page recording it as a hypervisor. Live guests recorded from a single pct list / qm list pass.

Finding 2 — four guests the wiki placed on pve live on m93p. CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are absent from live pct list on pve, yet every one of the four services answers. /etc/pve/lxc/ on pve contains only 260.conf.bak and no vzdump task log references CT 221/241/245/260 — so they were not deleted, they were never on this host. The 2026-10-11 inventory was wrong and every audit since carried it forward.

Finding 3 — the tailnet inventory had drifted badly. The page listed 30 nodes; live tailscale status reports 45 non-funnel nodes (plus 23 funnel-ingress-node IPv6 peers, now noted and deliberately excluded). Fifteen nodes were missing, including m93p, pve, omniroute-1, paseo, qwenpaw, rustdesk, vaultwarden, social-dashboard, openweb, omarchybook, sftpgo, rclone, omni and runtipi. Several states were wrong too (anthonys-mac-mini is online on a different IP; omarchy-macmini / omarchy-quattro are not live peers; anthonys-iphone is online, not active). The CT460 serve routes re-verified exactly as recorded — 13 routes plus :8443/webhook Funnel and :9443 tailnet-only — so that table needed no change.

Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling: camofox-browser.service now reports Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M) with 16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the fault, as diagnosed last cycle.

Also verified, unchanged: Mattermost CT255 stopped (peer offline 24d); VM 282 ubuntu-server-xfce is now running, not stopped as recorded.

Cron (verified from executions.db / jobs.json, 03:05): 2026-10-13 = 168 completed / 1 failed (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress 23 completed / 0 failed with the 03:00 stocktake 1a8e90ea2271 still running at probe time. Open incidents 3 — 1a8e90_d11698bd7275 (detected, 10-13 opencode 404), 0e753e_e7f8194a7d06 (detected since 10-01, next slot 2026-11-01), 1b23c0_c35703f637b8 (alerted since 10-09, next slot Fri 2026-10-16 08:00). Non-clean last_status: 4 (unchanged). Fleet 29 jobs — 26 enabled / 3 disabled.

Pages touched: infrastructure/proxmox.md (two-host rewrite, m93p inventory, four-entry correction, storage, VM 282), infrastructure/tailscale.md (45-node inventory + funnel-peer exclusion note, frontmatter), infrastructure/network-map.md (m93p entry, frontmatter), current-state.md (new Recent Changes block), index.md (proxmox/tailscale blurbs + header warning), systems/skills-index.md (stocktake artifact), raw/stocktake/freshness-20261014.txt (new), log.md.

Not verified, left as stale: the remaining 46 stale pages were not probed against a live source, so they stay flagged rather than rewritten. Not re-verified this pass and still single-host in framing: systems/omniroute, infrastructure/hosts, systems/model-providers, infrastructure/docker-services — m93p also runs its own omniroute (CT101), so the OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.


[2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node

Ran 2026-10-16 00:0x–00:3x AWST. Scope is two windows, not one: the 2026-10-14 23:59 Wiki Audit (cbfe434a3b5e) failed on the fault below, so the 10-14 window was never covered.

Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).

  • Root cause. httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101 (confirmed in config.yaml: custom provider name: omni-101, base_url: http://100.78.63.20:20128). It surfaced as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. — the exact string recorded on every failing execution row. 328 Errno 111 lines in the live errors.log, 504 across the rotations.
  • Impact. 10-14 = 160 completed / 12 failed; 10-15 = 151 completed / 24 failed (+1 in flight). Against 10-12 175/0, 10-13 167/1. 36 failure rows across 13 jobs, but only 24 are genuine model-call failures on 12 jobs — the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
  • The finding that matters most — there is no failover. Verified from live jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable pin, custom:omni (CT600, omniroute:20128), sits on e530c89f2be0 — which is no_agent: true; d40085da631c is disabled. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni. So zero model calls completed in 34 hours, and the fleet has no job that could even report the second node's health.
  • Correction made during this audit. A first pass read Evening Stand-Down Briefing's clean 18:45 runs (both days) as evidence CT600 stayed healthy. Wrong: it is no_agent and never calls a model. Recorded as a standing rule — never read a no_agent job's green row as provider-health evidence — in the incident page and in systems/scheduled-tasks.
  • User-visible cost: silence. Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The weekly Perth New Restaurant was missed outright (next slot 10-22).
  • Recovery, verified live (not assumed). 1b31187397a5 completed 10-15 22:01:56 (streak 4 → 0); 96eb95ce12f2 10-16 00:01:36; b63ed5817212 10-16 00:02:29; ea79a2107e36 10-16 00:09:48. Probes 2026-10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19.
  • Fleet bookkeeping. Non-clean last_status: 3 → 12 (9 new: 08eb86f02d37, ecc1fc13655a, dca8482e4f76, c8d52904e1d6, 06a0a1c8704f, 85edf6772bca, 0e519874d280, 1a8e90ea2271, cbfe434a3b5e; three cleared in the window: b63ed5817212, 1b31187397a5, ea79a2107e36). Open incidents 3 → 12 (9 alerted / 3 detected). Long-standing and pre-storm: 1b23c042e17d (Fri 08:00), 0e753e9720fd (1st 09:00), 066bb5191280 (delivery_failed, dead Telegram thread 34795).
  • Mnemosyne. Both monitors sit on the failing node, so no coverage figures were refreshed for ~34 h; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected.
  • Self-healing gap left alone deliberately. The 10-15 03:00 stocktake failure left raw/stocktake/freshness-20261015.txt unwritten and systems/skills-index.md stamped 10-14. Its next slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the scheduled run.
  • Sessions. 36 sessions across the two days, all cron, zero interactive — fourth consecutive window with no interactive session.

Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new, 9,303 B); systems/scheduled-tasks.md (frontmatter, Count current-state bullet, new Re-verified block); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + blast-radius note on the two monitors); current-state.md (frontmatter + new top section, Updated: line); index.md (Last updated, new incidents entry, scheduled-tasks bullet); log.md (this entry). Nothing removed or archived — no retired job, no obsolete system, so index.md structure gains one page.

Durable fix still open (needs Anthony's call): the single-node topology. All 18 model-calling jobs on one OmniRoute node means this recurs on the next omniroute-1 blip — as it did on 2026-10-09. Options are a second provider pin on a subset of jobs, or router-side failover. Not applied — that is a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.

[2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead

Window: cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran 2026-10-18 00:31 → 00:5x AWST. Sessions reviewed: 30, all cron, zero interactive — fifth consecutive window with no interactive session, so every finding below is from live system state, not conversation.

Summary

  • omniroute-1 (custom:omni-101) has a second failure mode: 503 load-shedding. 9 model calls across 7 jobs died on RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. between 10-16 08:00:37 and 10-17 03:00:28 (~19 h). The control plane stayed up the whole time (/api/health 200 {"status":"ok"}, /v1/models 401 — re-probed 10-18 00:33), so a green health endpoint is not evidence that the model path is serving. This is not a repeat of the 10-14 storm: that was reachability (Connection error., Errno 111), this was the router serving and refusing under load. → incidents/2026-10-16-omniroute-503-resource-pressure (new).
  • The burst ended on its own and the fleet is green again. 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01), 10-18 so far 7 / 0; no failure row of any kind since 10-17 03:00. Every affected job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31, ecc1fc13655a + ea79a2107e36 both 10-17 12:02, each auto-closing its own incident. No change was made on the node; nothing was restarted or repinned.
  • Mosquito Tracker removed — fleet 29 → 28 (1b31187397a5). Absent from jobs.json (updated_at 2026-10-18T00:36:17+08:00), last execution 10-16 15:00 (completed, delivered). The three disabled jobs are unchanged, so 25 enabled / 3 disabled. Non-clean last_status 12 → 8; only 1b23c042e17d (error, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
  • The watchdog double-counts. 96eb95ce12f2 exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one sentinel. Read failures per job, not per row.
  • Second consecutive stocktake hole, again left alone deliberately. The 10-17 03:00 slot failed, so raw/stocktake/freshness-20261017.txt is missing and systems/skills-index.md is stamped 10-16; 10-15 is missing too (storm). The 10-18 03:00 slot closes it — no manual regeneration, because a hand-run would race the scheduled one.
  • GitHub Trending Radar is dead at the delivery end. The Telegram bot token in its config.json returns HTTP 401 on getMe (probe 10-18 00:35); the job's GitHub PAT still returns 200, so only the delivery credential is gone. The cron row stays green (last_status from the 10-12 09:00 run, last_run_at 2026-10-12T09:00:12). Next exercise Mon 2026-10-19 09:00. Not repaired — a new bot token is Anthony's call, and the secret is not written here.
  • Incidents: 9 opened, 6 closed in-window; open 12 → 8 (5 alerted / 3 detected). The ledger files 7 of the 9 503 rows failure_type: unknown despite the explicit message (the two script-exit rows are typed script), so failure_type still cannot be used to triage.
  • Mnemosyne: both monitors rode the burst (ea79a2107e36 10-17 00:00, b63ed5817212 10-16 12:03), both clean on every slot since — including 10-18 00:00 (delivery_outcome: suppressed). No fresh coverage figures exist (they report to Telegram, not the ledger); last explicit read stays 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %), above the 95 % reindex threshold, so no reindex is due.

Pages touched: incidents/2026-10-16-omniroute-503-resource-pressure.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + stocktake row + Known Issues entry); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + new Re-verified section); current-state.md (frontmatter + Updated: line + new top section); index.md (Last updated, new incident entry, scheduled-tasks bullet, skills-index note); log.md (this entry). Nothing removed or archived — no page became obsolete in the window.

Bridge housekeeping (verified 2026-10-18): every page this run touched was byte-identical between the authoring tree (/home/hermes/workspace/agent-estate-wiki) and the vault copy (/home/hermes/wiki/agent-estate-wiki, pages nested under wiki/) before the edits, so the seven files were written to the authoring tree first, committed and pushed (4137dd6), then copied into the vault — sha256 matched on all seven. The vault-only pages (wiki/systems/ignis.md, wiki/systems/nextcloud.md) and the vault's own wiki/raw/stocktake/ series were not touched; nothing was deleted and no one-way sync was run. Drift found, not fixed: incidents/2026-09-02-sftpgo-outage.md exists only in the authoring tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.

Durable fix still open (needs Anthony's call): the single-node topology — all 18 model-calling jobs on custom:omni-101 with no failover, now seen failing two different ways in five days. Options remain a second provider pin on a subset of jobs, or router-side failover. Not applied — a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.

[2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST

Window verdict: one failure all day, and it was a failure mode the wiki had never recorded. Fleet 28 jobs (25 enabled / 3 disabled); 10-18 = 168 completed / 1 failed against 10-17 = 163 / 5; 10-19 so far 5 / 0 plus this audit. 11 sessions in the window, all cron — no interactive sessions, so no user decisions arrived to document; every finding below is machine-observed.

  • A model stream repeated itself and Hermes delivered the placeholder to Telegram. ecc1fc13655a (FreshRSS Midday, 12:00) wrote "Your request was not processed. Send it again if you still want me to carry it out.", delivered it (delivery_outcome: delivered, 12:08:11) and then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response. New page → incidents/2026-10-18-degenerate-stream-canned-substitute. The open question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue) instead of substituting text on the job's normal route?
  • Mattermost: cause found — CT255 is stopped, superseding the 10-11 entry's "cause not determined". Reconnect loop unchanged (attempt 471 at 00:14:07, connect timed out after 30s, 5-minute cadence), peer offline 29d, and the host answers neither TCP :8065 nor ICMP. Platform config still says enabled: true, so the loop will keep logging until the container is started or the block retired.
  • New page: systems/push-notifications — the bridge is broken and has been for as long as the logs go back. status.json: connected: false, error: …connection refused (nothing listens there on this machine); the bridge dials HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net, which /etc/hosts resolves to 127.0.1.1 where nothing serves 443, while the gateway itself is healthy on 0.0.0.0:9119 (/ → 302) and Serve still maps the name to it. 23,469 occurrences of that string across agent.log(.1-3) + errors.log(.1-2). Fix documented, not applied.
  • Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break. working_memory had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18 03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in the page); 1 unresolved conflict. The 24 h threshold in mnemosyne-health-check.sh:92 will keep flagging idle weekends.
  • Stocktake landed and closed the two-day hole: 1a8e90ea2271 exited 0 on 10-18 03:00, freshness-20261018.txt written, systems/skills-index regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (dd53e5f).
  • Carried forward, still open: the GitHub Trending Radar delivery token (getMe → 401, verified 10-18) — its next exercise is Mon 2026-10-19 09:00; the single-node custom:omni-101 topology with no model-call failover; and the vault drift below.

Pages touched: incidents/2026-10-18-degenerate-stream-canned-substitute.md (new); systems/push-notifications.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + FreshRSS row + totals); systems/messaging-integrations.md (frontmatter + new Mattermost section); systems/mnemosyne.md (frontmatter + new tier/health block); infrastructure/tailscale.md (frontmatter + inventory header + three missing peers + mattermost age); infrastructure/proxmox.md (CT255 age); current-state.md; index.md; log.md. Nothing removed or archived — no page became obsolete and no one-way sync was run.

Bridge housekeeping + vault drift (verified 2026-10-19 00:2x). Of the ten files this run touched, six were byte-identical between the authoring tree and the vault copy before the edits; two more (infrastructure/tailscale.md, infrastructure/proxmox.md) differed only because the vault holds older versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (wiki/ nesting) with sha256 verified per file.

Vault drift measured: against authoring HEAD, the vault holds 74 identical pages, 11 stale and 12 missing. Stale: infrastructure/network-map.md, infrastructure/tailscale.md, infrastructure/proxmox.md, infrastructure/docker-services.md, systems/skills-index.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, incidents/2026-08-01-mnemosyne-capture-failure.md. Missing: incidents/2026-09-02-sftpgo-outage.md (flagged by the 10-18 audit and still absent), the seven agents/* pages, reports/communities-wa-social-media-audit-2026.md, CONTRIBUTING.md, plus this run's two new pages (copied immediately after). The vault repo's own HEAD is still 9bacd8e mirror(2026-10-16) with a working tree carrying uncommitted 10-17/10-18 changes, and it has no remote, so it cannot be pushed. Not fixed — a full mirror pass is a bigger call than an audit's mandate; flagged here.

No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.

[2026-10-20] audit | Nightly wiki audit — omni-101 503 burst 2 (2026-10-19)

  • Incident re-opened. incidents/2026-10-16-omniroute-503-resource-pressure went back to status: active. A second, identical HTTP 503 … resource pressure burst hit omniroute-1 on 2026-10-19 — 9 model calls across 8 jobs (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 (2026-10-16 08:00 → 10-17 03:00). Whole day 159 completed / 13 failed; 2026-10-18 was 168 / 1.
  • Facts corrected from the source of truth (~/.hermes/cron/jobs.json, executions.db) instead of being carried forward: the burst-2 table's job names, schedules and model aliases were wrong (85edf6772bca is Tech & AI Newsletter Digest (Analytical), not "Fringe Festival Plan"; ea79a2107e36 is the Mnemosyne Health Monitor, not a news digest; ecc1fc13655a is FreshRSS Midday News Briefing), 08eb86f02d37 fails at 08:31 (not 08:01, it is a 30 8 * * * job), and 2026-10-17 reads 163 / 5, not 162 / 5. All eight failing jobs read provider: custom:omni-101, model: smart-route — the burst-1 note's wider alias mix (deepseek-paid-smart, free-IA, dark) no longer matches jobs.json and is now flagged in the page for confirmation rather than silently overwritten.
  • Incident ledger read properly for the first time. Unclosed rows = 13 (12 alerted / 1 detected), not the 7 the 10-19 entry recorded — the earlier figures used a narrower filter. A raw closed_at IS NULL count reads 12 at 10-19 00:20 and 13 now; the entry states the query so future entries compare like with like. 4 rows opened and 2 closed on 10-19, plus ea79a2_747b27556fd0 closed 10-20 00:04:24.
  • Pages updated: incidents/2026-10-16-omniroute-503-resource-pressure.md (burst-2 table rebuilt from jobs.json, new findings, status back to active), systems/scheduled-tasks.md (new current-count bullet, non-clean last_status rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected), systems/omniroute.md (new Recent Changes section), current-state.md (new Recent Changes section), index.md (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), log.md.
  • Nothing removed or archived — no page became obsolete and no one-way sync was run.
  • Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 15 sessions in the window, all cron — no interactive sessions, so there were no user-facing decisions to record.
  • No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
  • Vault drift (measured — sha256 per file against /home/hermes/wiki/agent-estate-wiki): 76 identical / 15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512 "missing", but those are build artefacts under public/, package*.json, serve.py and raw/stocktake/, which the vault deliberately does not carry.) All six pages touched by this audit were copied in with per-file sha256 verification. The stale pages left unmirrored — infrastructure/docker-services.md, infrastructure/network-map.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, systems/skills-index.md, incidents/2026-08-01-mnemosyne-capture-failure.md — are authoring-tree edits made from other agents' sessions; a full vault mirror pass remains the open call.
  • Not re-run this audit: the public/ Quartz rebuild and wiki-maintain.sh. Authoring commits 95fd7d3 → 766882e → 8da6ddf → 0248698, pushed to Gitea Anthony/agent-estate-wiki and confirmed at the remote.

2026-10-21 — nightly wiki audit (cron cbfe434a3b5e, window 2026-10-20)

  • A clean day — and a failure this ledger cannot see. 2026-10-20 closed 168 completed / 0 failed (plus 1 unknown), the first zero-failure day since 2026-10-12, ending a seven-day run. The omni-101 503 fault did not recur (last failure row 2026-10-19), so incidents/2026-10-16-omniroute-503-resource-pressure returns to monitoring. Zero incident rows opened, 4 closed → unclosed 13 → 9 (8 alerted / 1 detected).
  • New page: incidents/2026-10-20-gateway-unclean-exit-orphaned-run. The gateway died with no exit path at 2026-10-20 09:09:01 (last heartbeat 08:50:28, pid 2496376, suspected_oom=False), orphaning the 08:30 Gmail Action Digest run (08eb86f02d37, claimed 08:31:03) as unknown — "owner exited before a durable terminal state". No delivery row for the slot, no incident row created, and the job table still shows the previous slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits since 2026-08-01, 5 of them clustered on 2026-08-14/15).
  • New page: incidents/2026-08-08-gitea-config-backup-push-dead. e27da6c34bbb reads ok and its executions read completed nightly while nothing has reached Gitea since 2026-08-07 11:00:47 — 75 unpushed commits, fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'. Second instance of the "green row, dead job" pattern after the GitHub Trending Radar token revocation.
  • Corrected the 2026-10-18 canned-substitute page. Its "second surface" section attributed the interactive-session substitution to an HTTP 404 Model-not-found; the retained log evidence is Rebuilt-message restart limit (3) exceeded → reason=rebuilt_restart_limit_exceeded … api_calls=1/150 tool_turns=0 response_len=0. The page now carries the verified trigger and flags the unresolved model disagreement (session pin minimax/minimax-m3.1-flash:free vs model=qwen3.5:397b in the turn-end line). The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered, and is recorded as outstanding in current-state.
  • Pages updated: current-state (new Recent Changes section), systems/scheduled-tasks (new count bullet — non-clean last_status 12 → 9, config-backup row annotated, frontmatter), systems/current-profile (model default moved to nous / meituan/longcat-2.5-preview:free, no fallbacks key, pins unchanged), infrastructure/backups (the scheduled Gitea push has been dead since 2026-08-08), incidents/2026-10-16-omniroute-503-resource-pressure (status back to monitoring + dated update), incidents/2026-10-18-degenerate-stream-canned-substitute (trigger corrected), index.md, this log.
  • Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 20 of 28 pin OmniRoute; the 8 unpinned jobs moved upstream to nous with the global default change — the fleet now runs two upstream paths. 2 interactive sessions in the window (talaria-ios tax-return work; the desktop turn served the canned non-answer) — the first non-cron sessions after weeks of cron-only windows.
  • Nothing removed or archived — no page became obsolete, and no one-way sync was run.
  • No secrets written. The gateway token pasted in the 2026-10-20 desktop session was not transcribed into any page; both the incident and current-state entries describe the request without the credential. Host names, IPs, job/incident IDs, ports and probe status codes only.

[2026-10-21] stocktake | Scheduled wiki maintenance (cron 1a8e90ea2271, 03:00 AWST)

  • Maintenance script ran clean — wiki-maintain.sh completed all three steps: skills-index regenerated (346 → 347 skills; new australian-tax-return-review under Finance), freshness audit written (raw/stocktake/freshness-20261021.txt), Quartz rebuilt (596 files) and deployed to public/, hermes-wiki-serve.service restarted.
  • infrastructure/tailscale — node table rebuilt from live tailscale status --json. The 2026-10-19 table had drifted again: octop (100.92.163.122) and springroll (100.100.153.125) were missing entirely, openmausbot / openmausbot-vm were described in prose but absent from the table body, omarchymac had gone offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on DNS name rather than the JSON HostName — the latter is ct260 for the freshrss node, and both OmniRoute nodes share the hostname omniroute, which silently collides. Now 51 non-funnel nodes (50 IPv4 peers + hermes). Container states cross-checked against live pct list on pve (100.96.100.82) and m93p (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
  • reports/uber-eats — two missing monthly reports recovered. The generator (~/.hermes/cron/scripts/generate-uber-eats-report.py) wrote only to ~/wiki/reports/uber-eats/ (the ~/wiki repo), never to the canonical agent-estate-wiki tree, so the 2026-08 and 2026-09 reports were missing from the published wiki. Root cause fixed: the script now writes to ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree. It also now emits YAML frontmatter — without it the pages landed in the "no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing months copied in and verified against ~/.hermes/uber_eats_report.json (2026-09: $51.29 / 2 orders; 2026-08: $211.16 / 7 orders). The 2026-08 previous-month column was corrected — the original run had no comparison payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change.
  • incidents/2026-08-15-skills-index-drift and incidents/2026-09-02-sftpgo-outage re-verified against live state and annotated. The skills-drift page named a regenerate-skills-index.sh that no longer exists on disk (regeneration now lives inside wiki-maintain.sh step 1/3); the SFTPGo page confirmed CT299 running and the health endpoint returning HTTP 401 (auth required, service up) — no recurrence.
  • Still stale, left alone deliberately: reports/communities-wa-social-media-audit-2026 (54d) — its source document /home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md no longer exists on disk, so the audit baseline cannot be re-verified against a live source. Flagged, not edited.
  • Fleet unchanged — 28 jobs / 25 enabled. This job (1a8e90ea2271) is active, last_status: ok, model: free-stack / provider: custom:omni-101, next run 2026-10-22 03:00.
  • No secrets written. Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.

[2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the omni-101 pin

  • New page: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday, custom:omni-101 / smart-route) failed 2026-10-21 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — a string this fleet has never produced before, and not the 503 resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session cron_dca8482e4f76_20261021_070022 wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls); incident dca848_014affeb6954 is alerted and unclosed; it was the day's only failure (171 completed / 1 failed).
  • Infrastructure: Hermes Agent updated to 0.22.3 — hermes update ran 22:47:25 → 22:57:38 AWST (action 9eed3a99e799347c691ea52a72e122a1), pre 0.22.2 / d923a5eb… → commit 5b147ab036… (detached), stamp built 2026-10-21T14:57:38Z. Stage snapshot = skipped (disabled by updates.pre_update_backup (mode: off)), so the update had no snapshot and the off-box copy path is still dead. Gateway restarted via systemd (pid 868087, phase: running, heartbeat 00:13 AWST), dashboard respawned by argv (:9119, pid 869852). Advisories printed, neither actioned: rebuild the search index (hermes sessions optimize-storage) and Preparing pinned cua-driver… (no completion line in the log). The intermediate d923a5eb rebuild was never recorded on the page, so the previously-noted v0.21.5+6169.gab86a20 (2026-10-04) was already stale — corrected.
  • Pages updated: systems/hermes-agent.md (version/upstream bullets rewritten, frontmatter dated), systems/scheduled-tasks.md (new current-count bullet; 10-21 bullet marked superseded), systems/omniroute.md (new 2026-10-21 section: three signatures, one pin), incidents/2026-10-16-omniroute-503-resource-pressure.md (window-close update; stays monitoring), incidents/2026-08-08-gitea-config-backup-push-dead.md (2026-10-21 run: backlog 75 → 76, message_id 8449), infrastructure/backups.md (76 commits; pre_update_backup: false now has a concrete exercised example), current-state.md (new dated section), index.md (new incident entry, scheduled-tasks summary, date).
  • Ledger facts as recorded this window: fleet 28 jobs / 25 enabled / 3 disabled; non-clean last_status 9 (7 error / 2 delivery_failed) with one swap — 08eb86f02d37 out (its 08:30 slot ran clean and 08eb86_472c7b609c80 resolved 08:37:21), dca8482e4f76 in; 1 incident opened / 1 closed, unclosed stays 9 (8 alerted / 1 detected); 12 sessions, all cron, no interactive sessions; 2026-10-20 settled at 169 completed / 1 unknown (the orphan) — the 10-21 audit's "168 / 0" was taken before the 23:59 slot landed, and that correction is recorded in both systems/scheduled-tasks.md and current-state.md.
  • No secrets written. Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.

[2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)

  • Ran ~/.hermes/scripts/wiki-maintain.sh after the audit so the new incident page is actually published, rather than waiting for the 03:00 stocktake.
  • systems/skills-index.md regenerated from the live tree: 348 skills (was 347) across 74 top-level directories; section order now follows the script's category-priority list. Auto-generated — no hand edits.
  • Freshness report raw/stocktake/freshness-20261022.txt: 1 stale / 95 fresh / 6 no-date of 102 pages. The only stale page is reports/communities-wa-social-media-audit-2026.md (55 d).
  • Quartz build + deploy: 273 HTML files into public/; hermes-wiki-serve.service restarted 00:18:05 AWST (the build wipes and recreates public/, so the running server always needs the restart).
  • Verified live: http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip → 200, and all four of its wikilinks resolve (systems/omniroute, systems/scheduled-tasks, systems/hermes-agent, incidents/2026-10-16-omniroute-503-resource-pressure); the new entry appears twice on the served index.

[2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)

  • systems/scheduled-tasks.md — new ## Count bullet for the 2026-10-22 window: 172 completed / 0 failed (+ this audit running), zero failures for the second time in three days; fleet unchanged 28 jobs — 25 enabled / 3 disabled; five incidents closed in-window and none opened, so unclosed rows fall 9 → 4; non-clean last_status falls 9 → 5; 27 sessions (15 cron, 3 subagent, 6 desktop, 2 acp, 1 oneshot). Previous bullet marked superseded; updated/verified_on bumped to 2026-10-23, review_after to 2026-10-30.
  • current-state.md — new Recent Changes (2026-10-23) block at the top: the 172 / 0 window, incidents 9 → 4 unclosed and non-clean last_status 9 → 5, the 10-21 recount (168 / 1), the stocktake no-commit anomaly, the skills-index root fix, the Quartz deploy, and the session mix; updated/verified_on bumped to 2026-10-23, review_after to 2026-10-30.
  • Correction recorded: a DB recount (by both started_at and claimed_at, AWST) puts 2026-10-21 at 168 completed / 1 failed, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it (floor 2026-10-08); the delta is written into the page rather than silently changed.
  • incidents/2026-08-15-skills-index-drift.md reopened (status: resolved → active, reopened_on: 2026-10-23). Evidence: the 2026-10-22 stocktake (1a8e90ea2271) ran 03:00:28 → 04:38:19 (97 m 58 s, vs 16 m 01 s on 10-21), finished completed with delivery suppressed, and committed nothing — its regenerated systems/skills-index.md (mtime 03:00, ## .Archive (41) section, 347 total) was still uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo's wiki-maintain.sh (find-based, 347 with .Archive) and the stale ~/.hermes/scripts/wiki-maintain.sh (rglob-based, 348 without). Live counts verified: filesystem 348, with symlinks 376, minus .archive/ (41) and apple/ (4) = 331, loader catalogue 329 enabled (34 hub-installed / 52 builtin / 243 local).
  • Root-cause fix applied: the repo's wiki-maintain.sh now prunes */.archive/* from the skill scan (the loader does not expose archived skills), prints the archived count as an explicit exclusion, and names hermes skills list as the authoritative loadable figure. systems/skills-index.md regenerated: 306 skills across 69 categories (was 347 counting the parked archive). Script fix and regenerated page committed together.
  • Quartz rebuild + deploy: 102 files parsed, 602 files emitted to public/, deployed, and hermes-wiki-serve.service restarted 00:14:22 AWST.
  • Freshness report raw/stocktake/freshness-20261023.txt: 2 stale / 94 fresh / 6 no-date of 102 pages — the long-standing reports/communities-wa-social-media-audit-2026.md (56 d) and, newly over the threshold, systems/model-providers.md (22 d, updated 2026-10-01). The latter needs a review pass against the live OmniRoute catalogue before its next verified_on.
  • No secrets written. Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.

[2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST

  • New page: systems/conduit.md — the Conduit dashboard plugin + iOS app (voice calls, cards, push). Plugin tree ~/.hermes/plugins/conduit_push, version 0.64.0 (HEAD eaa14be, no tags upstream), remote kaishi00/hermes-conduit-notifier; gateway on :9119; one registered device (iPhone, app 2.0.0); state/DB files and sizes recorded. Where the wiki previously had no page for the phone surface at all.
  • New incident: incidents/2026-10-22-conduit-memory-start-timeout — three Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call warnings (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s MEMORY_TIMEOUT_S in the plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree to a 60 s default (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that degrades instead of aborting; the dashboard restarted 18:54:43 and no abort line exists after it (two calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). The patch is uncommitted.
  • incidents/2026-10-09-cron-provider-timeout-storm.md → status: resolved. Its last two rows (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b, both on 1b23c042e17d) closed 2026-10-23 08:04 when that weekly job's Friday slot ran clean; the job reads ok again. All 12 rows closed — this incident no longer contributes an unclosed row anywhere.
  • Cron ledger: window 2026-10-23 closed 169 completed / 0 failed (+ this audit running) — the third zero-failure day in four; 2026-10-22 settled at 173 / 0. Fleet 28 jobs (25 enabled / 3 disabled), unchanged for a sixth audit. Unclosed incident rows 4 → 2 (c0a9fa_764558b5ae83 alerted since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and 0e753e_e7f8194a7d06 detected since 10-01, next slot 11-01). Non-clean last_status 5 → 4 (c0a9fa314542, 0e753e9720fd, fad5a86979d4, 066bb5191280). Pins unchanged (11 omni-101 / smart-route, 8 unpinned, 6 omni-101 / free-stack, 2 custom:omni / smart-route, 1 deepseek-paid-smart = this audit). 14 sessions: 13 cron + 1 desktop (18:26:12, the Conduit phone-call session that exposed the fault above).
  • systems/mnemosyne.md: new block recording the 10–45 s cold start and the consumer-timeout rule (a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold start itself was not optimised — only the budget was raised.
  • Also updated: systems/scheduled-tasks.md (new Count bullet + ## Re-verified 2026-10-24 section), current-state.md (new Recent Changes block; 10-23 block marked superseded), index.md (date, new systems/conduit and incident lines, refresh of the scheduled-tasks / skills-index / storm blurbs), and frontmatter added to incidents/2026-07-29-mnemosyne-relay.md and incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md — both sat outside the SCHEMA (rule 4: every page has frontmatter).
  • Freshness: 102 pages scanned at the last stocktake; 3 carry a verified_on older than 21 days — infrastructure/hosts.md, runbooks/provider-health.md, runbooks/update-hermes-safely.md (all 94 d, unchanged). Nothing removed or archived this cycle: no job retired, no system obsolete, no decision reversed in the window.
  • Note (hygiene): a second, older copy of this wiki sits at /home/hermes/wiki/agent-estate-wiki (101 pages, 3 incidents, no git remote, last commit c798304 "mirror(2026-10-22 maintenance)"). It is not served — hermes-wiki-serve.service serves public/ from this repo — so it is a stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than syncing it, since no script owns it.
  • No secrets written. Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.

[2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused

  • Window 2026-10-24: 167 completed / 0 failed (+ this audit running); 2026-10-23 settled at 170 / 0 (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet 28 jobs (25 enabled / 3 disabled), 4 non-clean last_status, 2 unclosed incident rows (c0a9fa_764558b5ae83 alerted since 10-19; 0e753e_e7f8194a7d06 detected since 10-01) — nothing opened, nothing closed in-window. Pins unchanged. 14 sessions: 11 cron + 3 desktop.
  • New incident page: incidents/2026-10-24-conduit-call-request-refused.md. Anthony asked to be phoned three times on 2026-10-24 12:53–12:56 (Call me now to test calls. → Yes. You do → CAL ME); every reply asserted the agent cannot make calls, with 0 tool calls in the session. conduit_call_user exists, is registered for every session, is discoverable via tool_search, and has a bundled skill (calling-the-user); the call settings were permissive (enabled / when_asked / decides / alerts all true). Root cause: capability asserted instead of discovered — not a missing tool, and not one weak model (the free model's answer was repeated after a switch to smart-route).
  • Correction to incidents/2026-10-22-conduit-memory-start-timeout.md: its closing evidence (“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a false positive. conduit-calls.json → ends is a ring buffer of turn-ends that fire a watch (MAX_RECENT_ENDS = 64), and its two entries are the cron sessions of that night (Mnemosyne Embedding Coverage Monitor 00:01, Health Monitor 00:04). history — real call records — is empty, so no call has been placed since the fix and the 60 s budget is unexercised by a call. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43 restart.
  • systems/conduit.md refreshed: version 0.64.0 / eaa14be → 0.69.0 / f000bf0 (22 commits on); the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the uncommitted plugin_api.py patch is now five releases behind HEAD; new section documenting the conduit_call_user tool, its gate conditions and rate limits; device/state figures re-verified (last_seen 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59); store 569.9 MB / 1,520,229 events / 3,028 buckets — the 10-24 page's 308 buckets / 62 windows was a narrower grouping, corrected rather than read as a collapse.
  • Also updated: systems/scheduled-tasks.md (new Count bullet + ## Re-verified 2026-10-25 section + a date-bucketing note), current-state.md (new Recent Changes block; 10-24 block marked superseded), index.md (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22 line).
  • Freshness: not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last run: 102 pages scanned, 3 carry a verified_on older than 21 days (infrastructure/hosts.md, runbooks/provider-health.md, runbooks/update-hermes-safely.md — all 94 d, unchanged). They are the next cleanup candidates. Nothing removed or archived this cycle: no job retired, no system obsolete, no decision reversed in the window.
  • Method note recorded in systems/scheduled-tasks.md: executions.db stores started_at with a +08:00 offset and SQLite's date() normalises to UTC, so date(started_at) moves 00:00–07:59 AWST rows onto the previous day. Bucket days with substr(started_at,1,10).
  • No secrets written. Job IDs, incident IDs, table names, counts, paths and timestamps only.

[2026-10-25] stocktake | wiki-maintain.sh + drift correction (tailscale, homepage-dashboard, 2 stale pages)

  • Script ran clean. wiki-maintain.sh completed all three stages at 03:02–03:03 AWST: systems/skills-index.md regenerated (302 skills / 69 categories, 45 archived — the count fell from 306 because four skills moved to ~/.hermes/skills/.archive/: courier-tracking-monitor, pve-vm-console-recovery, model-provider-comparison-research, live-application-verification — all four confirmed present in the archive directory), freshness audit written to raw/stocktake/freshness-20261025.txt, Quartz rebuilt (630 files) and deployed to public/, hermes-wiki-serve.service restarted. Verified the served pages return 200 afterwards.
  • Freshness audit at run time: 2 stale / 99 fresh / 4 no-date of 105 pages. The two stale pages were both re-probed live and corrected (see below); after correction the audit reports 0 stale.
  • infrastructure/tailscale.md — node table rebuilt from live tailscale status --json. The 2026-10-21 table had drifted again:
    • springroll-browser (100.109.221.91) was missing from the table entirely — added (online).
    • omarchybook (100.107.9.100) and openmausbot (100.91.214.91) have both gone offline (online → idle/offline 3d). openmausbot's m93p LXC (CT1264) is still running but no longer has the tailscale binary on PATH, so the node itself has stopped connecting.
    • Every offline age was stale and was refreshed: mattermost 31d → 35d, omni 64d → 68d, elementsynapse 91d → 95d, ubuntu-server-xfce 60d → 64d, cloudreve 88d → 92d, anthonys-imac 27d → 31d, cachyos 47d → 51d, conduit 44d → 48d, freshrss-new 64d → 68d, omarchymac 3d → 7d.
    • homeassistant still advertises ExitNodeOption: true but ExitNode: false — it offers exit-node capability but is not currently serving as one; the old "active; offers exit node" wording overstated the live state and was corrected.
    • m93p (100.104.51.98) reads idle/offline 13m ago in the peer table, but the host is plainly reachable (both pct list probes and the SSH session succeeded during this audit), so it is recorded as a Tailscale-side liveness artefact, not a hypervisor outage.
    • Container identities confirmed the authoritative way this run: tailscale ip -4 executed inside each candidate LXC on both pve (100.96.100.82) and m93p (100.104.51.98), because peer names and LXC names do not always match. CT209 (cloudreve), CT255 (mattermost), CT555 (elementsynapse) and CT601 (omni) confirmed stopped; CT600 (omniroute, live OmniRoute) and CT101 (m93p) confirmed running.
    • Funnel peer count re-checked: still 23 funnel-ingress-node peers with IPv6-only addresses (excluded). Total non-funnel nodes 51 → 52 (51 IPv4 peers + hermes itself).
  • infrastructure/tailscale.md + systems/homepage-dashboard.md — the CT201 route tables were wrong. Live pct exec 201 -- tailscale serve status shows CT201 serving Tailscale on both :443 and :8443, and the old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead" note was false: no Traefik container or process exists on CT201 (docker ps -a and ps aux both come back empty for traefik), and ss -ltnp shows tailscaled bound to 100.96.244.39:443. Corrected routes: / → 8085 and /mgr → 8084 on both ports; /steel → 5173 and /steel-api → 3000 on :443 only (/steel on :8443 returns 502; on :443 it returns 200). The old table also routed :8443 straight to http://localhost:8082, which has no listener — the live target is 8085 (asset_proxy.py), which itself proxies to the dead 8082 and returns 502. Both pages corrected. The CT460 route table was checked and matched live exactly (12 paths, no drift).
  • reports/communities-wa-social-media-audit-2026.md (was 58d stale) — re-verified, not drifted. Checked against the live Department of Communities pages: the remit wording, the ten portfolios, the 1 July 2025 Housing → Department of Housing and Works restructure, the five-minister structure and the WA Seniors Card carve-out all still hold; the Department site was updated 23 October 2026 and its recent announcements (carers, youth grants, forced adoption) match the content pillars recorded here. Audience metrics were not re-counted and are left as the 2026-08-28 point-in-time figures; review_after: 2026-11-26 stands.
  • incidents/2026-08-26-work-outlook-basic-auth-disabled.md (was 22d stale) — re-probed, still open. himalaya envelope list -a work -s 1 reproduces the identical failure (cannot build IMAP client / login not supported). The tenant still advertises only XOAuth2 and the account still authenticates by password, so no remediation has landed. Status stays open; updated and verified_on bumped to the verification date with the probe output recorded on the page.
  • systems/scheduled-tasks.md checked and left unchanged. Live ~/.hermes/cron/jobs.json reads 28 jobs — 25 enabled / 3 disabled, matching the page. All 28 job IDs are present on the page. Non-clean last_status is still 4 (0e753e9720fd error, fad5a86979d4 delivery_failed, 066bb5191280 delivery_failed, c0a9fa314542 error streak 1) and unclosed incident rows are still 2 (0e753e_e7f8194a7d06, c0a9fa_764558b5ae83) — identical to the page's 2026-10-25 entry. No drift, nothing to change.
  • No secrets written. Job IDs, incident IDs, IPs, counts, paths and timestamps only.

[2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential

  • Fleet was quiet but one job was not. Live executions.db: 2026-10-25 settled 172 completed / 2 failed; both failures are 08eb86f02d37 (Gmail Action Digest), which also fired six times in the day (08:31:14 → 10:26:37 runaway at 1 h 55 m vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53). New signature on the 19:18:41 fire: Cron admission missing: the owner never admitted this fire. All six fires — failures included — carry delivery_outcome: delivered. Neither failure created a cron_incidents row; the page's open-incident count is unchanged at 2. → new incidents/2026-10-25-gmail-digest-runaway-and-refire.md
  • Root cause of the missing context compression found: the Nous Portal credential is dead. Nous Portal runtime credentials unavailable: Invalid refresh token — 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04, agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. → new incidents/2026-10-22-nous-portal-refresh-token-invalid.md
  • ⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value. A 2026-10-24 19:06:10 switch note reads "model was just switched from meituan/longcat-2.5-preview:free to smart-route via omni-101"; live config confirms custom:omni-101 / smart-route (api_mode: chat_completions), config.yaml last written 2026-10-25 19:13. A six-provider top-level fallback_providers chain is live again (headed by the dead nous credential), so the 2026-10-21 "no fallback chain" record is stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → the fleet is a single upstream path again. Corrected on systems/current-profile.md, systems/model-providers.md, concepts/model-routing.md, comparisons/model-routers.md, systems/omniroute.md.
  • Auxiliary routing captured. At 2026-10-25 19:13 all eight auxiliary: slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off nous to no-think/freeapi/auto on custom:omni-101 with free_only: true and a freeapi/deepseek-v4-flash → kg/kilo-auto/free chain. ⚠ Masking, not fixing — nous remains first in fallback_providers. New section on systems/model-providers.md.
  • systems/scheduled-tasks.md — new 2026-10-26 entry added; the 2026-10-25 entry marked superseded. Fleet unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean last_status 4 → 5 (08eb86f02d37 joined as delivery_queued with an empty last_error); pins unchanged.
  • systems/conduit.md re-probed live. Version 0.69.2 in plugin.yaml and dashboard/manifest.json (both), HEAD 942ddb8 (2026-10-24 15:35:42, "Pulse: list candidates, decide in code, journal them (#342)"), 0 behind origin/main, 24 commits on since the 0.64.0 baseline, working tree still carrying the uncommitted dashboard/plugin_api.py patch. The dashboard gateway restarted 2026-10-25 18:42:33, so the upstream commits are now live — the previous "not live" note is superseded. conduit-calls.json → history is still empty; conduit_show ran once on 10-25 14:37. Status on incidents/2026-10-24-conduit-call-request-refused.md stays open (untested, not fixed).
  • systems/messaging-integrations.md — CT255 still stopped (Mattermost offline 36 d, tx 312 / rx 0); 268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not outage duration).
  • systems/hermes-agent.md — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04, 19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.
  • index.md — header date, the two new incident links, and rewritten conduit, messaging-integrations and scheduled-tasks summaries.
  • No secrets written. Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
  • Not changed: current-state.md (its dated sections are historical records), and the dead custom:omni provider entry (still named by 2 jobs — Anthony's call, not the wiki's).

2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)

wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, hermes-wiki-serve.service restarted). The first freshness report listed 8 stale pages; each was then diffed against live state and corrected where reality could be verified. A second script run then rewrote the same-day report with 0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with.

  • infrastructure/hosts.md — CT600 is live, not retired. Live pct list on both hypervisors shows CT600 running on pve (100.88.81.19) and CT101 running on m93p (100.78.63.20); CT601 does not exist on either host (pct config 601 → no such config), so the old "retired → CT601" note was wrong on both counts. CT333 (clawtest) moved from "unverified" to verified running. Both hypervisors now carry a verified row (pve 100.96.100.82 / 17 LXCs; m93p 100.104.51.98 / 19 LXCs). Config version 33 → 51; skills 66 → 302; cron "25 active" → 28 (25 enabled / 3 disabled); Xvfb display :102 → :99 (Xvfb :99 -screen 0 1600x1200x24). The omni provider entry was replaced by the two real ones — custom:Omniroute:20128 → http://omniroute:20128/v1 (6,480 models) and custom:omni-101 → http://100.78.63.20:20128 (the live default provider). The authoring tree vs the vault mirror are now distinguished by path.
  • agents/hermes-production.md — Tailscale IP corrected: 100.96.100.82 → 100.118.5.51. The old value is the PVE hypervisor, not CT460. Version v0.21.5 → v0.22.3 (5b147ab036d6); stable v0.22.5 is available (hermes update --check). The dead 8082 port claim was removed (no listener; live ports are 9119 gateway + 8085 webhook). Default provider is custom:omni-101, and the fallback chain was rewritten from live config — its first entry is the dead nous credential, so a fallover tries a broken provider first. All eight auxiliary: slots are no-think/freeapi/auto on custom:omni-101 (moved 2026-10-25 19:13). Cron 30 (27/3) → 28 (25/3) with the 5 non-clean last_status named. Mattermost still unreachable — CT255 stopped, peer offline 36 d.
  • agents/hermes-chaos.md — version v0.22.3, and both CT450 and CT460 now run the same build (5b147ab036d6); the "3 commits behind" claim is superseded (0 behind origin/main). Default model is smart-route on custom → http://100.78.63.20:20128, not the old deepseek-flash-mix on custom:omniroute. Cron "11 paused" → 25 jobs (13 enabled / 12 disabled), with 9 names shared with CT460 (8 disabled here, Important-mail monitor the reverse).
  • infrastructure/tailscale.md — node table rebuilt from tailscale status --json (keyed on DNSName, IPv4-filtered) and container identity re-confirmed from inside each LXC via pct exec <vmid> -- tailscale ip -4 on both hypervisors. octoagent (100.117.10.68) and omarchymac-1 (100.126.173.115) were missing entirely and are now listed — 52 → 54 non-funnel nodes (53 live IPv4 peers + hermes). m93p now reads online (it was a stale peer record). Every offline age was refreshed (mattermost 35d → 36d, omni 68d → 69d, elementsynapse 95d → 96d, ubuntu-server-xfce 64d → 66d, omarchymac 7d → 8d, omarchybook 3d → 4d, openmausbot 3d → 4d). The omni row now reads "CT601 removed from both hypervisors" rather than "CT601 stopped". Serve tables re-probed per host: CT460 matched live exactly (12 paths on :443 plus the :8443 Funnel and :9443 route); CT201 still 502s on / (asset proxy → dead 8082) while /mgr and /steel return 200 — the 502 claim was re-confirmed, not merely carried over.
  • systems/memory-backend.md — re-verified against live config.yaml: added reflect.disabled_for_cron, sleep_threshold: 50, nudge_interval: 10, and live totals (2,350 memories — 842 working / 1,090 episodic / 418 memoria_facts). Embedding coverage re-measured: 1,899 / 2,350 (80.8 %) overall, working 815 / 842 (96.8 %) and episodic 1,084 / 1,090 (99.4 %) — both tiers above the 95 % reindex threshold, so no reindex was triggered.
  • runbooks/provider-health.md — the step-3 OmniRoute snippet was broken and raised IndexError: the provider was renamed omniroute → Omniroute:20128 and the key moved from an inline api_key to a key_env indirection. The snippet now resolves the key from key_env. A new step 3b documents the omni-101 quirk — its /v1/models returns Invalid API key even with the correct key while /v1/chat/completions works (a smart-route call returned minimaxai/minimax-m3.1-flash), so a models check on that node is a false failure. Also corrected the typo "config slocated" → "located". Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI /health 200, opencode/nous/ollama fallbacks all 200.
  • runbooks/update-hermes-safely.md — added the managed-runtime dependency re-assert step (hermes-runtime-deps-guard.sh) after an update, since the updater can leave a runtime without fastembed/dotenv/mnemosyne. Clarified --branch main vs the stable channel (currently v0.22.5, local v0.22.3). Version and last_tested brought to 2026-10-26.
  • incidents/2026-10-03-ct450-hermes-update.md — added a Current state section: both instances now run v0.22.3 (5b147ab036d6), CT450 is 0 behind origin/main, and the CT450 Mattermost timeout is still an infrastructure condition (CT255 stopped), not an update regression.
  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md — added a Current state section: the follow-up is satisfied — fastembed 0.8.1 imports in both managed runtimes (19da945341ad4794ac2bc8ff9fdd3643, d41b16c8b6674454a4a819bd4acced1f) and 0.8.0 in the legacy venv, FASTEMBED_CACHE_PATH pinned. Current coverage figures recorded; both tiers above the 95 % threshold, so monitor job b63ed5817212 (last_status: ok) has nothing to fire on.
  • All 9 corrected pages verified at http://127.0.0.1:9120/<path> — every one returns 200, and the served infrastructure/tailscale contains octoagent, omarchymac-1 and 54 non-funnel.
  • No secrets written. Job IDs, incident IDs, base URLs, key env var names, counts, paths and timestamps only.

Vault mirror drift (2026-10-26, measured not eyeballed)

The vault at /home/hermes/wiki tracks 125 files under two conventions: agent-estate-wiki/<path> (28 files — the Obsidian-visible tree the recent audit commits write) and agent-estate-wiki/wiki/<path> (93 files — the fuller mirror).

This run forward-synced every page it touched into both conventions where the directory existed — 22 copies, all sha256-verified against the authoring source before the vault commit. Two vault copies differed from the authoring tree (infrastructure/tailscale.md, systems/skills-index.md under wiki/); both were confirmed to be older versions (2026-10-21 vs authoring HEAD 2026-10-25), so no direct vault edits were at risk.

Remaining drift (left for an explicit request — a full mirror pass is bigger than an audit's mandate):

Convention match stale missing
agent-estate-wiki/<path> (top-level) 25 2 80
agent-estate-wiki/wiki/<path> 34 58 15

The two top-level stale files are current-state.md and reports/communities-wa-social-media-audit-2026.md — both dated historical records rather than drift. The vault repo is local-only (no remotes); it was committed, not pushed.

[2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run

  • A planned gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and this time it was loud. gateway.log: Received SIGTERM as a planned gateway stop at 14:33:35 (under_systemd=yes, parent_name=systemd, loadavg_1m=2.15). The drain then logged Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True → interrupting remaining work, and at 14:34:29 Skipping .clean_shutdown marker — drain timed out with interrupted agents. The victim was ea79a2107e36 (Mnemosyne Health Monitor), claimed 12:01:13 — 2 h 34 m in. It closed 14:35:59 failed on a string this wiki had not seen: RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership. Unlike the 10-20 orphan it raised two cron_incidents rows (14:33:37 ea79a2_c3ab2d784ba3 "Cron execution unverified; reconcile admissions/724a69108b…json"; 14:35:59 ea79a2_a4661554a3c2, the RuntimeError) and delivered the failure. → new incidents/2026-10-26-planned-restart-orphaned-cron-run.md
  • The Mnemosyne Watchdog did its job. 96eb95ce12f2 red-flagged twice (14:46:12, 15:01:15, Script exited with code 1) reacting to the same log ERROR, then self-resolved at 15:16:12 when the line aged out of its 15-minute window. Both rows (96eb95_0c89bd373a0a, 96eb95_e32174134546) are closed.
  • Window tally and fleet: 2026-10-26 settled 169 completed / 3 failed, and all three failures are this one interruption (Health Monitor + the two watchdog reds); 2026-10-25 settled 172 / 2. Fleet unchanged: 28 jobs, 25 enabled / 3 disabled. Non-clean last_status 5 → 4 (c0a9fa314542 and 08eb86f02d37 cleared to ok; ea79a2107e36 joins as error, streak 2). Unclosed incident rows 2 → 3. Pins unchanged. One gateway start (14:35:18, teardown 54.06 s). 17 sessions in window (15 cron, 1 ios, 1 telegram).
  • One incident row closed — the 2026-10-16 503 burst is fully shut. c0a9fa_764558b5ae83 auto-closed 09:13:00 when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on 10-25 or 10-26. Status stays monitoring (the single-node exposure is unchanged). Updated incidents/2026-10-16-omniroute-503-resource-pressure.md.
  • ⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread. (a) The radar's 07:00 slot closed completed at 09:10, but its delivered text read “RUN INCOMPLETE — tool-call budget exhausted” — no Notion writes, no cover letters — so the day's backlog was executed by hand. (b) fad5a86979d4's 03:08 slot completed but ended delivery_failed: Thread 34795 not found — the same dead thread the gateway's own 14:33:37 shutdown notice hit. → runbooks/job-radar.md
  • The radar's cover-letter pipeline changed the same morning. The PDF is now printed from the canonical rendered HTML via Chromium (build_cover_letter_pdf_from_html.py, created 10:37) instead of ReportLab, so a download matches the on-screen letter; the generation default moved to deepseek-paid-smart (10:47) and job_dashboard_server.py's regenerate timeout went 120 s → 600 s (10:48, server restarted). → runbooks/job-radar.md
  • Files: new incidents/2026-10-26-planned-restart-orphaned-cron-run.md; updated incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md, incidents/2026-10-16-omniroute-503-resource-pressure.md, runbooks/job-radar.md, systems/scheduled-tasks.md, current-state.md, index.md, log.md. Nothing removed or archived. No secrets written.
  • Unresolved: why a planned stop bypasses the drain deadline (timed_out=True at 1.35 s of a 30 s budget); whether failure_streak should count the unverified row (one interruption currently reads streak 2); and the Thread 34795 delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the unthreaded chat.

[2026-10-28] audit | Nightly wiki audit — window 2026-10-27 00:00 → 2026-10-28 00:2x AWST: the radar's tool-call budget ran out a second time — and this time the run is recorded as failed

  • Window 2026-10-27: 172 completed / 1 failed (+ this audit running). The failure is the radar's budget overrun, and it is the story of the night → new page incidents/2026-10-27-job-radar-budget-exhausted.
  • The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is recorded as failed. dca8482e4f76 ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closed failed and delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery, verification, dedup and classification completed; Notion ingestion, cover-letter generation and the final cover-letter audit never ran. Notion DB 0f90ba2b-8b10-4d02-a62c-6f692d5b1168 held 451 entries with 0 added for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88). Incident row dca848_e93a538ade6b opened 11:03:53, alerted 11:03:59, still open at this audit (failure_type: auth — which does not describe a budget fault). The previous day's identical overrun closed completed and was found only by reading the delivered text — that is why the 10-27 row matters.
  • Response, same day: the radar is now two jobs. Stage 1 dca8482e4f76 Daily Marketing Job Radar - Discovery (0 7 * * 1-5, deliver: local, failures → Telegram topic 1030) does DISCOVERY → VERIFY → DEDUP → SCORE, writes a handoff file and triggers stage 2. New stage 2 5fc292ff0f3b Daily Marketing Job Radar - Write and Letters (30 11,14 * * 1-5, Telegram topic 1030, created 2026-10-27T17:34:15) lands the Notion entries, cover letters, missing-letter audit and report. Two stage-2 test fires on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both completed, delivery suppressed, last_status: ok, streak 0. Pins unchanged (custom:omni-101 / smart-route). Fleet 28 → 29 jobs (26 enabled / 3 disabled). → runbooks/job-radar.md
  • The 2026-10-26 orphan class did not recur; its two incident rows closed. ea79a2_a4661554a3c2 and ea79a2_c3ab2d784ba3 auto-closed 2026-10-27T00:38:32+08:00 — unclosed rows fleet-wide 3 → 2 (the 2: 0e753e_e7f8194a7d06 Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new dca848_e93a538ade6b). No 10-27 execution row carries the orphan strings and the watchdog alert log has no 10-27 entry. → incidents/2026-10-26-planned-restart-orphaned-cron-run.md
  • Non-clean last_status 4 → 5 (churn): ea79a2107e36 cleared to ok; dca8482e4f76 joined as error (streak 1) and 08eb86f02d37 re-joined as delivery_queued. Also non-clean: 0e753e9720fd (error), fad5a86979d4 (delivery_failed), 066bb5191280 (delivery_failed).
  • Two config corrections from live probes. tts.provider is openai pointed at the Mac mini speech server (https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech, model phonon, voice alloy) — the page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked superseded rather than deleted). Live voice uses voice.speech_server.url = wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/; GET /v1/pool → {"size":2,"in_use":0}, both units idle, while /v1/models returns {"detail":"Not Found"}. voice.auto_tts is still false, and wake_word.enabled is true. → systems/hermes-agent.md, systems/conduit.md
  • systems/skills-index.md was regenerated by the 2026-10-27 03:00 slot but never committed — the authoring tree was dirty with updated: 2026-10-27 and a 302 skills / 69 categories total (against index.md's stale 306 / 70 of 2026-10-23). Freshness at that run: 0 stale / 104 fresh / 4 no-date of 108 pages (freshness-20261027.txt). Committed, and index.md corrected to 302.
  • infrastructure/tailscale.md peer table re-verified against tailscale status --json; the node-count header and any state changes are recorded in that page.
  • Files: new incidents/2026-10-27-job-radar-budget-exhausted.md; updated incidents/2026-10-26-planned-restart-orphaned-cron-run.md, runbooks/job-radar.md, systems/scheduled-tasks.md, systems/hermes-agent.md, systems/conduit.md, infrastructure/tailscale.md, systems/skills-index.md, current-state.md, index.md, log.md. Nothing removed or archived. No secrets written.
  • Unresolved: whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30 cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified failure_type: auth; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.

[2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted

  • New page: incidents/2026-10-28-job-radar-stage1-iteration-cap — stage 1 (dca8482e4f76) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed failed, failure_streak: 2, on "hit the agent iteration cap before the handoff could be written". No handoff file existed (~/.hermes/radar-ref/handoff/ held only 2026-10-27.json), so both stage-2 slots (5fc292ff0f3b, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) completed with delivery_outcome: suppressed — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a different ceiling (tool-call budget on 10-27 vs iteration cap on 10-28).
  • Updated: incidents/2026-10-27-job-radar-budget-exhausted (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did not close, unclosed rows 2 → 4) · runbooks/job-radar (10-28 slot write-up; "first real cycle" marked as run-and-empty) · systems/scheduled-tasks (new 2026-10-29 window bullet; prior bullet demoted to Prior window; the two in-window job rows refreshed) · systems/browser-backend (dated correction: Camofox restored, browser-steel plugin removed, endpoint dark, STEEL_BASE_URL residue in 10 live processes) · comparisons/steel-vs-camoufox · decisions/rollback-steel-to-camofox (2026-10-29 update) · systems/skills-index (stocktake-run note) · current-state.md (new 2026-10-28 section) · index.
  • Stocktake folded, not paged: 1a8e90ea2271 ran 03:00:28 → 03:37:58, closed failed (streak 1, delivery suppressed) on RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203 after regenerating systems/skills-index.md (302 skills) and raw/stocktake/freshness-20261028.txt (2 stale / 103 fresh / 4 no-date of 109 pages); it left both uncommitted and filed incident 1a8e90_f57ab4cafab8 in state detected with no alert. The audit committed the two artefacts, clearing the dirty tree.
  • Live probes (2026-10-29 00:3x AWST): jobs.json 29 jobs / 26 enabled / 3 disabled, 5 non-clean last_status, pins 12 smart-route / 6 free-stack / 7 unpinned / 1 deepseek-paid-smart; executions.db — 2026-10-28 settled 168 / 2 / 1 running, 2026-10-27 now 173 / 1, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP :9223 and REST :3000 refuse, tunnel /steel 502, steel.martinwa.org unresolved; no Hindsight service on this box (:8888 closed, no hindsight units) so systems/memory-backend and comparisons/memory-backends were left unchanged.
  • Removed/archived: none. Secrets written: none.
  • Unresolved: the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); 1a8e90_f57ab4cafab8 sits detected without an alert; both radar incident rows close only on a clean 07:00 weekday slot.

[2026-10-29] update | Steel Browser fully removed from the estate — Camofox is the sole backend

  • Root cause of the long-standing cloud_provider: steel residue: the 2026-07-30 rollback disabled the browser-steel plugin and set browser.engine: camofox, but left browser.cloud_provider: steel in ~/.hermes/config.yaml. Every browser tool call then routed through _instantiate_explicit_cloud_provider() and failed with "no registered browser plugin has that name: steel". Because engine was already correct, the split config was inert-but-wrong and survived ~3 months of audits.
  • Hermes (CT460): browser: block now reads exactly cloud_provider: camofox, camera_provider: camofox, engine: auto, cdp_url: "". The browser-steel plugin directory was deleted, and the browser-steel key was removed from both plugins.disabled and plugins.entries. STEEL_BASE_URL was dropped from ~/.hermes/.env. Hermes core source contains zero Steel references, so there is no Steel fallback path in the tree — returning to Steel now means redeploying it.
  • CT201 (runtipi): the entire Steel deployment was torn down — containers steel-browser-api-1 / steel-browser-ui-1, the steel-network Docker network, the images steel-browser-api:fixed and both ghcr.io Steel images, and the project dir /root/steel-browser (~6 GB reclaimed; disk 70G → 64G used). The Tailscale Serve routes /steel (5173) and /steel-api (3000) were deleted with tailscale serve --set-path <path> off; CT201's remaining routes are only / → 127.0.0.1:8085 and /mgr → http://127.0.0.1:8084, on both :443 and :8443. Nothing listens on 3000, 5173 or 9223.
  • CT333 (clawtest): a second, separate and orphaned Steel install was found and removed — the steel-browser container (ghcr.io/steel-dev/steel-browser:latest, 5.53 GB, created 2026-05-30, publishing port 3000) and its image. It had no Serve route, no entry in ~/.zeroclaw/config.toml (which uses backend = "agent_browser" on 127.0.0.1:9515), no reference from Nanobot's config or the CLI-anything browser harness; its last CDP navigation was 2026-10-20, it received no requests after 2026-10-21, and its page crashed 2026-10-22.
  • Estate-wide sweep: every LXC on the PVE host was checked — no Steel containers, images, networks or listeners remain anywhere. Homarr, the homepage dashboard and the runtipi app configs had no Steel references.
  • Gateway restarted 01:21 AWST to clear STEEL_BASE_URL from the inherited process environment; the new gateway tree (PID 3913841) reads STEEL_BASE_URL=0, and Telegram, Discord, ntfy, email and Home Assistant all reconnected cleanly. The dashboard unit and agentmail-webhook.service still carry the variable in their inherited environment, but no core code reads it.
  • Reversibility preserved: CT201's compose file and Dockerfile are archived under /root/_archived-steel/, plus /root/_archived-steel/steel-cache-archive-20261029.tgz; the images remain re-pullable from ghcr.io. Backups: config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z, config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z, .env.bak-pre-steel-cleanup-20261028T220947Z.
  • Camofox verified healthy after every change: REST http://100.118.5.51:9377 (and http://localhost:9377), /health → 200, /tabs → {"running":true,"tabs":[]}, and live navigation confirmed working.
  • Pages corrected (15): systems/browser-backend · decisions/rollback-steel-to-camofox · decisions/index · comparisons/steel-vs-camoufox · comparisons/camoufox-vs-chrome · infrastructure/docker-services · infrastructure/tailscale · infrastructure/hosts · systems/homepage-dashboard · concepts/browser-automation · runbooks/recover-docker-service · current-state · index · agents/nanobot · systems/scheduled-tasks.
  • Corrects the 2026-10-29 00:3x audit entry, which described Steel as merely "not reachable / endpoint dark". It is removed, not down. Per SCHEMA rule 9 the superseded claims are marked, not deleted.
  • Removed/archived: none. Secrets written: none (variable names only; no key or URL values).
  • Unresolved: the obsolete CT333 line in Nanobot's Obsidian vault (Areas/Homelab/architecture.md, "Steel Browser | Clawtest | http://localhost:3000") still stands — that vault is outside this wiki and was not modified. Also noted, unrelated to Steel: Browser Use viewing unavailable: unsupported Hermes browser resolver has logged in the gateway since 2026-10-11.

[2026-10-29] stocktake | General Agent Estate Wiki maintenance — Tailscale table re-verified, and a 23-day-old "never re-authenticated" claim turned out to be false

  • Script: wiki-maintain.sh ran clean (exit 0). systems/skills-index.md regenerated from the live skill tree — 302 skills in 69 categories, with a new note that 45 skills are parked under ~/.hermes/skills/.archive/ and excluded from the count. (303 SKILL.md files exist outside .archive/, but productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md is a nested alias duplicate of productivity/obsidian-git-sync/SKILL.md, so 302 unique skills is the correct figure.) Freshness report written to raw/stocktake/freshness-20261029.txt; Quartz rebuilt (110 pages → 644 files) and deployed to public/.
  • Freshness report: 2 stale / 105 fresh / 3 no-date of 110 pages. The 2 stale were incidents/2026-10-01-cliproxy-connection-auth-expired.md and incidents/2026-10-05-dead-model-pin-repin.md, both at 23d.
  • Updated infrastructure/tailscale — State column rebuilt from tailscale status --json; 11 rows changed. Two were state transitions: sftpgo (CT299, pve) is back online (the 2026-10-28 audit had recorded it offline <1d; tailscale ping → pong in 0s, systemctl is-active sftpgo → active, tailscale ip -4 inside the guest → 100.88.223.70), and m93p is online despite a stale peer record (Online: false in the JSON, but tailscale ping → pong in 1ms and SSH over the tailnet returned m93p / up 11 days) — the second occurrence of that pattern after 2026-10-26. The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged at 54 (53 live IPv4 peers + hermes); JSON reports 76 peers — 53 with a 100. IPv4, 23 IPv6-only funnel ingress. No missing peers, no IP mismatches, CT601 still absent from both hypervisors. Container identity re-confirmed from inside all 34 guests via pct exec <vmid> -- tailscale ip -4 on both hypervisors; the 3 stopped guests (CT209/CT255/CT555) and the 3 running-but-unconnected guests (CT105/CT1264/CT100) confirmed.
  • Updated incidents/2026-10-01-cliproxy-connection-auth-expired — the residual claim "the connection credential itself was never re-authenticated" is falsified. A live POST /v1/chat/completions for cliprox/terra against custom:Omniroute:20128 (CT600) returns HTTP 200 — the connection has since been re-authenticated, independently of the incident's remediation. The same pin on custom:omni-101 (CT101) still returns HTTP 401 (No active credentials for provider: cliprox.), so the two nodes carry separate credentials. Claim marked superseded rather than deleted (SCHEMA rule 9); the remediation itself (moving 3 jobs off the pin) was never reversed and remains correct.
  • Updated incidents/2026-10-05-dead-model-pin-repin — every claim re-probed live: all 7 jobs are still on custom:omni-101 with routing aliases (free-stack ×6, smart-route ×1) ✔; both aliases return HTTP 200 ✔; ollama-cloud/qwen3.5:397b still HTTP 410 (retired 2026-09-25) ✔; nvidia/deepseek-v4-flash still HTTP 400 (absent from the live catalog) ✔; command-code/gpt-5.6-terra now returns HTTP 401 (credential) rather than the original 403 (plan-gated) — the pin is still dead, only the failure signature changed. Frontmatter updated/verified_on bumped to 2026-10-29 with created left at 2026-10-06 as the data period.
  • Updated systems/scheduled-tasks — the Known-Issues entry repeating the "never re-authenticated" claim now carries the correction. The page's fleet figures were re-checked against live jobs.json and remain accurate: 29 jobs / 26 enabled / 3 disabled, 5 non-clean last_status.
  • Verified, no change needed: systems/homepage-dashboard (CT201 ss -ltn shows 8084 and 8085 listening, no 8082 listener, Homarr on 8102 — all as the page claims); CT201 Serve routes (/ → 8085, /mgr → 8084 on both :443 and :8443, exactly as documented, Steel routes still gone); CT460 Serve routes (12 paths on :443 + :8443/webhook Funnel + :9443, matching the page).
  • Not changed, flagged for a human: the vault mirror at /home/hermes/wiki/agent-estate-wiki/ has drifted on both conventions — top-level 35 match / 3 stale / 72 missing, wiki/ 41 match / 53 stale / 16 missing. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
  • Removed/archived: none. Secrets written: none (credential names and HTTP status codes only — no key or token values).
  • Unresolved: cliprox/terra remains 401 on CT101 — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.

[2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed

  • New page — incidents/2026-10-29-conduit-voice-silence: every Conduit voice call connected and then died silently. Three faults sat in series, each one masking the next: (A) voice.speech_server.url was the truncated wss://…:8443/v1/, which the plugin dials verbatim, so the Mac answered 403 and the plugin blamed the key; (B) dashboard.basic_auth.session_ttl_seconds was the 12 h default, so an expired phone session got WS 4401 with no Hermes turn and no error on the device (the iPhone re-authenticated at 08:00:59, minutes after a silent call); (C) both host providers were dead — tts.provider: openai → Mac /v1/speech 502, stt.provider: groq → 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → …:8443/v1/realtime, TTL → 2 592 000 s (30 d), stt.provider: local (faster-whisper base), tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by two live calls at 08:38:04 and 08:49:36 — the first entries ever in conduit-calls.json → history, so the memory fix is no longer evidence-free.
  • Updated systems/conduit — new runtime-checks section, the two real calls on record, the incident cross-reference, and verified_on: 2026-10-30.
  • Updated systems/hermes-agent — new dated section 2026-10-30 00:0x AWST for the voice path (fault table + why it stayed invisible + the two-TTL-keys warning + backups), and the tts.provider block now carries a second superseded note recording the live values (ollama_local / local, /v1/realtime).
  • Updated systems/scheduled-tasks — new Count bullet for the 2026-10-30 window, the previous bullet demoted to superseded, stage 1's delivery column corrected to local (verified live: stage 1 no longer delivers), both radar rows annotated with their 2026-10-29 result, and a new Resolved 2026-10-29 block under Known Issues.
  • Updated runbooks/job-radar — new dated slot section 2026-10-29 07:00 (handoff written → consumed → delivered: 170 discovered / 33 duplicates avoided / 37 verified; 20 pages + 2 cover letters, missing_cover_letter_count 0, Notion board 455 → 475), plus the note that the second real cycle completed.
  • Updated incidents/2026-10-27-job-radar-budget-exhausted and incidents/2026-10-28-job-radar-stage1-iteration-cap — both resolved. Stage 1 ran 07:00:59 → 10:16:12 (3 h 15 m 13 s), closed completed, and both ledger rows it had opened (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed completed at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed.
  • Updated current-state, index — new Recent Changes block and the refreshed page descriptors.
  • Cron health: window 2026-10-29 settled 176 completed / 0 failed — the first zero-failure day since 2026-10-24 — with 10 deliveries, all 10 delivered. Fleet unchanged at 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3 (all three are stale re-runs of long-standing rows) and every failure_streak reads 0. Unclosed incident rows 4 → 1, the lone survivor being the Facebook monthly's 0e753e_e7f8194a7d06, which cannot clear before its 2026-11-01 slot. Pins unchanged.
  • Vault mirror catch-up: the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this audit were mirrored into /home/hermes/wiki/agent-estate-wiki/, including systems/conduit.md, runbooks/job-radar.md, systems/scheduled-tasks.md, systems/hermes-agent.md, current-state.md, index.md and log.md. The wiki-maintained pages under infrastructure/ remain a separate, known gap.
  • Removed/archived: none. Secrets written: none — provider names, key names and HTTP status codes only, never a key or token value.

2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state

  • Ran wiki-maintain.sh. Regenerated systems/skills-index.md (302 skills / 69 categories — unchanged; the generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote raw/stocktake/freshness-20261030.txt (0 stale / 109 fresh / 3 no-date of 112 pages), rebuilt Quartz (654 files) and deployed to public/.
  • Freshness is not accuracy. The age scan found nothing, but a live-vs-page diff against the running estate found real drift on five pages. All corrections below were verified against live sources before being written.
  • infrastructure/tailscale — node table rebuilt. tailscale status --json now reports 50 non-funnel nodes (49 live IPv4 peers + hermes), not the 54 the page claimed. 2 IP changes (anthonys-mac-mini 100.74.17.15 → 100.76.60.37; omarchymac 100.84.180.69 → 100.126.173.115 — the peer formerly listed as omarchymac-1) and 4 nodes dropped out of the tailnet entirely (anthonys-imac, cachyos, omarchymac-1, omni). Every offline age refreshed. m93p reads Online: false for the third consecutive audit yet is plainly up (tailscale ping → pong, SSH → up 12 days), so the row records it online. Total peers 72 (49 IPv4 + 23 IPv6-only funnel). CT460's :443 route set matched the page exactly; CT201 serves only / → 8085 and /mgr → 8084 on both :443 and :8443.
  • systems/scheduled-tasks — fleet ledger corrected. Live jobs.json shows 29 jobs (25 enabled / 4 disabled), not the 28/25/3 on the page: ea79a2107e36 (Mnemosyne Health Monitor) was auto-paused 2026-10-30 01:04:22 after its 00:01:08 slot closed unknown on a gateway-connection failure. Non-clean last_status is 4 (not 3), and the standing "every failure_streak reads 0" claim was false (ea79a2107e36 = 1). "Failing (delivery only)" is 2, not 1 (fad5a86979d4 joined 066bb5191280). The retired DHL row was dropped from the job table (it lives in Removed Jobs). Seven job rows carried stale error claims for jobs that have since cleared to ok — each was marked superseded with a dated correction rather than rewritten (SCHEMA rule 9). The executions.db retention floor note was corrected to the live 2026-10-24 09:22:36. Window tallies: 2026-10-29 = 177 / 0 and 2026-10-28 = 169 / 2 (both +1 on the page — the 23:59 Wiki Audit row had still been running at audit time).
  • New incident page: incidents/2026-10-30-mnemosyne-health-monitor-paused. The Mnemosyne Health Monitor's own failure notice failed to send (send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed) — deliveries.db holds zero rows for 2026-10-30, so the incident row is the only trace. The job was not re-enabled and the gateway was not restarted by this run; root cause is unverified and recorded as such.
  • infrastructure/hosts — four corrections. CT201's LAN address is 192.168.178.77, not 192.168.178.166; CT450's Hermes Agent is v0.22.5 (2026.9.24), local bf0203a794a0 — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both (a stale root-level /usr/local/bin/hermes on CT450 still reports v0.19.0 and is not the live agent — the live one is the hermes-user install under ~/.hermes/hermes-agent); the cron count is now 29/25/4; and the orphaned omni peer has expired out of the tailnet entirely (no peer named omni, no peer at 100.93.204.4), so the "offline 69d" description no longer applied. The skills count of 302 was left alone — it is correct.
  • systems/homepage-dashboard — five corrections, and a second pass to fix my own first pass. "All Apps" is 114 → 123 live apps, not 122 (the API dedupes by (name, normalised href) across groups; the literal All Apps group in services.yaml holds 122 and is not what the tab bar counts). The grid is a responsive repeat(auto-fill, minmax(230px, 1fr)) (served /dash.css line 3640), not a fixed 5-column grid. The tab list was wrong — but my first correction was wrong too and had to be redone: the live tab bar is built from the service groups (renderTabs() iterates state.groups), not from the layout: block in settings.yaml. That block and its tab:/columns: keys are inert — grep -c "Overview" /dash.js = 0 and grep -c "columns" /dash.js = 0. There is no "Overview" tab; the live tabs are ☆ Pinned, All (123) and the 14 groups (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20, Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved theme is {"pair":"Boarding Pass","mode":"light"} — one of 59 theme pairs; its light-mode amber is #795000, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. manager.py runs inside the homepage-manager container (its container Cmd is the pip install … && python3 /opt/homepage/manager.py line, confirmed via docker inspect and docker top); only asset_proxy.py is a host process (0.0.0.0:8085 users:(("python3",pid=4038))). And the "no Traefik container exists" evidence was wrong — the runtipi-reverse-proxy Traefik container exists but is stopped (Exited (255) 2 months ago); the conclusion that Traefik does not occupy 443 stands. The same evidence error was corrected in infrastructure/tailscale.
  • current-state and index updated with the new incident and the corrected fleet numbers.
  • Verified, not assumed: CT600 running and CT601 absent from both hypervisors (pct config 601 → no such config on pve or m93p); the CT201 :443/:8443 Serve tables; CT201 listeners 5173/3000/8082 (none — Steel's removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row (read from jobs.json, executions.db — whose cron_incidents table lives inside it, the standalone cron_incidents.db/incidents.db files are empty stubs — and deliveries.db).
  • Not changed: systems/skills-index.md count (302 is correct), CT460's Serve route table (matched live exactly), and the steel-removal records across the wiki (confirmed still true).
  • Secrets written: none — provider names, key names and HTTP status codes only, never a key or token value.

[2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST

Verdict: one large undocumented event found and written up, and one documented incident found to have been recorded wrong.

  • NEW PAGE — incidents/2026-10-30-ct460-host-outage-scheduler-gap. CT460 was dark for 5 h 30 m on 2026-10-30 and no monitor alerted. Evidence: zero executions.db rows between 10:58:20 and 16:37:21; PVE host pve boot time 2026-10-30 16:24:00; CT460 boot 16:28:00; the previous gateway life (pid=3913841, started 2026-10-28T17:21:58Z) logged UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"); scheduler resumed 16:37:21 with an 8-run catch-up burst through 16:38:22. The day still closed 138 executions / 0 failed against 10-29's 177 / 0 — ≈39 lost slots, no failures. Root cause not established: no OOM evidence locally or on the PVE host, so the ledger string is read as a candidate list, not a diagnosis. Still open: the two jobs that could have detected it (cbfe434a3b5e NFS mount monitor, 1a8e90ea2271 m93p LXC boot monitor) are disabled.
  • CORRECTED — incidents/2026-10-30-mnemosyne-health-monitor-paused was right about the record and wrong about the run. ea79a2107e36's 2026-10-30 00:01:08 slot was still running at the 03:3x reading; it closed completed at 10:02:48 after 10 h 01 m 40 s and delivered, auto-closing incident ea79a2_d93c3acdc4f2 resolved at 2026-10-30T10:02:48.662+08:00. Live job state is now enabled: true, last_status: ok, failure_streak: 0. The page's "zero deliveries.db rows / the notice was never delivered" claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with 8 delivery rows, all delivered (latest 18:46:04). Page frontmatter moved active → resolved, with the old snapshot kept below as explicitly historical. ⚠ Its 12:01 slot was lost to the host outage and is the still-running 16:37:21 catch-up. Reconciliation lesson: a slot mid-flight is not a failure — check finished_at, not last_status, before writing an incident.
  • Reconciled — systems/scheduled-tasks.md. Fleet 29 jobs (26 enabled / 3 disabled) — the "4th disabled job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean last_status 4 → 3, failing 1 → 0, the health-monitor table row, the disabled table row, and a new "Resolved 2026-10-30" subsection). Non-clean now 3 = 1 error + 2 delivery_failed; unclosed incident rows 2 → 1 (survivor: the Facebook monthly's 0e753e_e7f8194a7d06, due 11-01).
  • Reconciled — current-state.md (new leading cron entry + superseded marker), infrastructure/hosts.md (cron row), index.md (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries), incidents/2026-10-30-mnemosyne-health-monitor-paused.md.
  • Checked, unchanged: 12 sessions active in-window (longest local-40472e0b…, 1,351 messages), no orphaned running rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.

[2026-11-01] audit | Nightly review — Conduit 0.122.2, the Local Live fault, a force-drained reload, dead Asana paths

  • NEW PAGE — incidents/2026-10-31-conduit-voice-local-mode-not-applying. The app's "local" voice mode never applies even though its connection test succeeds — because the test checks socket reachability only. Every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint → HTTP 501, "start the server with --enable_llm_proxy"), /v1/usage reads llm_proxy.requests: 0 across 25 observed turns, and its realtime socket accepts the connection but rejects session.update as an unknown event. voice.mode is a phone-side selector: no mode key exists in config.yaml → voice or conduit-settings.json (whose only device state is devices/<UUID>/mode: small ×3), so "the change did not apply" cannot be diagnosed on host config. Evidence from session 20261031_193709_40b9c2 (started 19:37:09, 30 msgs / 17 tool calls, still open at the 00:2x reading); voice.local_live.front_model was repointed to mistral/voxtral-small-latest mid-session (config.yaml mtime 21:02:30). Nothing was changed on the Mac. ⚠ One misread was caught in-session and is recorded as a lesson: local_live.py:882 is the only writer of _local_live_last_request, inside the chat-completions route — so a "recent last_request_at" was the investigation's own probe, not phone traffic (last_request_at is null).
  • NEW PAGE — incidents/2026-10-31-gateway-reload-force-drain-orphaned-run. The 18:06:41 gateway reload waited the full 1,800 s drain cap and force-drained at 18:37:38 (new gateway 18:38:43), orphaning ea79a2107e36 (Mnemosyne Health Monitor, 12:01:06 slot*, 6 h 38 m 09 s in) which closed failed at 18:39:15 on "lost its durable fire claim ownership". Derivative watchdog reds at 18:39:23 / 18:45:23 / 19:00:22 cleared 19:15:21. A second, clean restart followed at 19:23:39 → 19:24:40. Same class as the 10-26 planned-restart orphan — second instance, first on a plain reload.
  • NEW PAGE — incidents/2026-10-31-asana-access-failures. Both Asana paths on CT460 are non-functional: the MCP server asana (https://mcp.asana.com/v2/mcp, auth: oauth, enabled: true) has never authenticated (~/.hermes/mcp-tokens/ holds only asana.client.json, 319 B, mtime 2026-10-26 01:07; logs read "MCP OAuth setup failed for 'asana' … no cached tokens found. Run hermes mcp login asana interactively first"), and the hosted connector rejects the call shape. The repair needs an interactive login, which cron cannot perform.
  • UPDATED — systems/conduit.md (13,184 → 18,722 B; frontmatter updated / verified_on → 2026-11-01). Version 0.69.2 / 942ddb8 → 0.122.2 / 9eac857 (10-26 reading kept explicitly superseded); the ⚠ "uncommitted patch, five releases behind" warning is RESOLVED — the tree is clean and MEMORY_TIMEOUT_S = 10.0 ships upstream in dashboard/api/memory.py (with a wait_for backstop and tests/test_memory.py), so the 10-23 in-tree edit is no longer needed; dashboard gateway pid 4042003 (10-29 08:03:26) → pid 176771 (10-31 19:35:02); the call-store shape change is recorded (v: 1; history empty; last_request_at: null; ends = 2 × outcome: stopped at 21:06:05 / 21:10:48; settings.quiet_at_screen: true = #479); data volume 930.0 MB → 938.2 MB (983,793,664 B) with 2,623,879 events / 4,384 buckets / 504 windows (≈8 MB/day, no retention policy); new dated section; Open items — patch item marked resolved, three new local-live items added.
  • UPDATED — systems/scheduled-tasks.md (same pass): fleet 29 jobs (26 enabled / 3 disabled); 2026-10-31 closed 160 / 4; non-clean last_status 4 (2 error + 2 delivery_failed); failing jobs 1 (ea79a2107e36, failure_streak 2); superseded readings marked rather than overwritten.
  • UPDATED — systems/hermes-agent.md: Asana MCP auth-state re-verification note (enabled but never authenticated) + both 2026-10-31 gateway restarts (18:06:41 force-drain, 19:23:39 → 19:24:40 clean).
  • UPDATED — index.md: frontmatter and banner → 2026-11-01; Conduit entry (version, patch resolved, gateway pid, new fault, empty call history); three new incident entries added above the 10-30 pair.
  • UPDATED — log.md: this entry.
  • Checked, unchanged: ship rows before 10-30 (nothing new); inc_20261031_000001 (gateway-update) is still open, inc_20261030_000001 closed 2026-10-30 18:05; 3 deliveries.db rows for 2026-10-31 carry a failure error (child run failed: exit status 1) — they are the failing runs' own notices, not new faults; no page archived; no secrets, tokens, passwords or keys written.
  • Probes (context, not findings): omniroute-1 100.78.63.20:20128 → HTTP 200; m93p 100.104.51.98 and pve 100.96.100.82 answered neither ICMP, SSH nor :8006 from CT460 at the 00:2x reading — recorded as unverified reachability from this container, not as a host outage.
  • Mirror catch-up (a whole-tree pass, beyond this audit's own pages). Because the previous copy bridge only ever synced the pages an audit touched, the user-facing vault had silently drifted. This pass re-copied every committed page in the authoring tree into both vault layouts (/home/hermes/wiki/agent-estate-wiki/ and .../wiki/) — 61 pages — which closed the 9-page drift and picked up everything the 2026-10-31 audit had changed and never mirrored (current-state.md, incidents/2026-10-30-ct460-host-outage-scheduler-gap.md, incidents/2026-10-30-mnemosyne-health-monitor-paused.md, infrastructure/hosts.md, systems/mnemosyne.md) plus 53 older stragglers. Deliberately left unsynced: the 6 files the 2026-10-31 03:00 stocktake has left uncommitted in the authoring repo (infrastructure/docker-services.md, infrastructure/proxmox.md, infrastructure/tailscale.md, systems/messaging-integrations.md, systems/omniroute.md, systems/skills-index.md) and untracked raw/stocktake/freshness-20261031.txt — the mirror convention is "(authoring COMMIT_SHA)", so uncommitted content stays out until the stocktake's own commit lands.

[2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago

  • BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake). wiki-maintain.sh exited 1 at step 3/3 with an esbuild panic. Root cause was not esbuild: incidents/2026-10-31-conduit-voice-local-mode-not-applying.md had an unquoted YAML title: containing the substring : (from "voice mode: local"), so YAML parsed it as a mapping entry and Quartz aborted with bad indentation of a mapping entry (2:27). A scan of all 117 pages found exactly one offending frontmatter line; the title is now double-quoted with escaped inner quotes. This is a latent trap worth remembering: any future page title containing ": " or ": unquoted will silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather than a YAML error.
  • STALE PAGE — incidents/2026-10-09-memory-recall-relevance.md (23d) — RESOLVED, not stale. The page's central claim was that the recall-ranking fix "was reported saved and is not present in live config", and that conflict 152 was "the only open conflict". Both claims are false, and the page was the reason the incident looked open for three weeks. Live ~/.hermes/mnemosyne/config.yaml reads vec_weight: 0.5, fts_weight: 0.3, importance_weight: 0.3 (plus MNEMOSYNE_IMPORTANCE_WEIGHT=0.3 in the environment) — the weights live in Mnemosyne's own config file, not in ~/.hermes/config.yaml's memory.mnemosyne block that the 2026-10-10 probe read. The 2026-10-10 verification probed the wrong file and concluded "not applied". The original regression test now passes: mnemosyne recall "Are you up to date" returns only substantive current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode (id 32f469aae09d56dc, still in the DB) no longer outranks anything. Conflict 152 was auto-resolved 2026-10-29 23:53:47Z ("low-confidence LLM extraction noise") — it has not been open since. All three open items closed; page moved to status: resolved. Live tier counts at 2026-11-01: working 941, episodic 1,124, canonical 90, conflicts 175 with 11 open (ids 165–175, all one-batch extraction noise sharing a single fact_a_id), working embeddings 927/941 (98.5 %), episodic 1,124/1,124 (100 %) — both above the 95 % floor, no reindex needed.
  • UPDATED — systems/mnemosyne.md: stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01 block added, including the weights' actual home and the resolved-incident pointer.
  • UPDATED — infrastructure/tailscale.md: table rebuilt from live tailscale status --json — 1 state transition and 5 offline-age drifts, no IP changes, no missing peers. omarchymac (100.126.173.115) went offline (Online: false, LastSeen 2026-10-31 13:20Z, tailscale ping times out — a genuine transition; the JSON OS field reads linux). m93p's peer record reads Online: false again — the 4th consecutive stale record — while tailscale ping → pong and SSH returned up 1 day, 12:02, so the row stays online. Ages refreshed: cloudreve 98→99d, freshrss-new 74→75d, omarchybook 9→10d, openmausbot 9→10d, ubuntu-server-xfce 71→72d; conduit (55d) and elementsynapse (102d) were already correct.
  • UPDATED — infrastructure/docker-services.md: CT201's Tailscale IP was wrong — the page read 100.116.116.116; live pct exec 201 -- tailscale ip -4 returns 100.96.244.39 (which [[infrastructure/hosts]] already had right). The CT200 inventory had three misspelled overtchat-* entries (written overchat-*) and one stale entry, worldmonitor-seeder-fast, which is not running — the list is now regenerated verbatim from live docker ps. CT333's first container was renamed: adoring_jemison → thirsty_payne. Counts re-verified: CT200 61, CT201 20, CT333 8, CT1001 1 (sshwifty), CT460 ignis running.
  • UPDATED — infrastructure/proxmox.md: storage percentages refreshed against live pvesm status on both hosts (pve fred-qnap 82.45→82.77 %, SSD1 63.80→64.32 %, local-lvm 13.60→13.75 %; m93p nfs-proxmox-* 82.45→82.78 %, omb-internal 58.00→57.92 %). pct list / qm list on both hosts re-confirmed — no CT601 on either hypervisor, CT255 still running, VM 103 cachyos-m93p stopped, VM 106 openmausbot-vm running. m93p uptime recorded as up 1 day, 12:02 with a load average of 7.66 / 12.80 / 11.32 — elevated, worth watching.
  • UPDATED — systems/omniroute.md / systems/messaging-integrations.md: re-probed today — both OmniRoute nodes answer 401 on /v1/models (alive, auth required), CT600 and CT101 both running; Mattermost's systemctl is-active → active, LISTEN *:8065, /api/v4/system/ping → HTTP 200. Frontmatter verified_on bumped to 2026-11-01.
  • REGENERATED — systems/skills-index.md: 301 unique skills in 68 categories, 46 archived. The count is unique (category, skill_name) pairs, not SKILL.md file count — live find returns 302 files, and the one difference is the known nested alias duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md vs productivity/obsidian-git-sync/SKILL.md. 301 is correct — do not "fix" it to 302. One skill was archived since the last run: hermes-runtime-worker-diagnostics (46 archived, up from 45).
  • FRESHNESS: the 2026-11-01 report lists 1 stale / 113 fresh / 3 no-date of 117 pages. The single stale page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after the fixes, per the script → fix → re-run → commit order.
  • Probes (context, not findings): m93p SSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (tailscale ping → pong throughout). homarr on CT201:8102 returns 307 → 200 on /board, so the dashboard page's "live, 200" claim still holds. CT201's :8443/ root returns 502 (upstream 8082 has no listener) and :8443/mgr returns 200 — both still as the homepage-dashboard page records. No secrets, tokens, passwords or keys written.

[2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by -15, and a red status with no row behind it

Window: 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on state.db (sessions), ~/.hermes/cron/{jobs.json,executions.db,deliveries.db}, gateway.log / agent.log.1, and both git trees.

  • NEW — incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md (status: active): the Sunday 18:00 79f9bb41ecf5 closed unknown at 18:05:13 with "Gateway connection failed; no local fallback", delivered nothing, and its incident row 79f9bb_9c443b602eec is still alerted. deliveries.db has no row after 11:01:12. Batch-mate b63ed5817212 hit the identical error and self-healed in 68 s. The same page records the inverse silent failure: ecc1fc13655a reads error, last_run 11-01T12:01:00, failure_streak: 1 — with no execution row and no incident row.
  • NEW — incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md (status: resolved): three workers SIGTERMed (-15) after adopting their claim (85edf6772bca 07:30:46; 96eb95ce12f2 + 0e753e9720fd 09:02:18) → "whether side effects ran is unknown". Fourth variant of the orphan class, and the first with no gateway restart/reload/drain anywhere in the window (PID 175269 up since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a 30 s worker-ack window vs a ~70 s cold start (Plugin discovery complete: 83 found, 69 enabled). All three incident rows resolved; one execution left unknown.
  • UPDATED — systems/scheduled-tasks.md: ## Count advanced to the 2026-11-02 window (167 completed / 1 running / 2 unknown vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the first non-clean day in four); fleet 29 jobs / 26 enabled / 3 disabled unchanged; unclosed incident rows 1 → 1 but a different row (all three rows the 10-31 audit left open closed on 11-01, including 0e753e_e7f8194a7d06, open since 2026-10-01 — 31 days); new ## Re-verified 2026-11-02 section covering the three failure modes, the 09:00 cold-start pair and the ledger. Non-clean last_status 4 → 2 (ea79a2107e36, 79f9bb41ecf5).
  • UPDATED — systems/messaging-integrations.md: Telegram degraded all day (dual-stack api.telegram.org ConnectTimeout → IPv4-literal fallback; disconnect 20:51:53, polling restarts 22:33:30 / 23:12:23), and getent hosts currently returns only an AAAA record for api.telegram.org — consistent with, but not proof of, a systematic timeout on the path DNS hands the client. The relay platform has had no adapter for ~29 h (Reconnecting relay attempt 147 @10-31 18:39:24 → 336 @11-02 00:10:47, counter never reset — itself the proof of no gateway restart). 066bb5191280's thread_id 34795 home-DM miss is the second confirmed variant of the thread-mismatch bug: the thread-less fallback fires, the content lands, and the run is recorded failed anyway. Day's delivery health: 8 rows / 7 delivered / 1 failed.
  • UPDATED — index.md: updated/verified_on → 2026-11-02, the audit summary line rewritten for this window, and the two new incident links added to the incident list.
  • Not done / unchanged: no page removed or archived; no job added, removed, disabled or repinned; no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts below instead of a bulk copy) — an audit's mandate is smaller than a full sync.
  • DRIFT (context, not findings): the authoring tree was clean and in sync with Gitea (origin/master..HEAD empty) before this audit; touched pages were mirrored into the vault's existing conventions and hash-verified. No secrets, passwords, tokens or keys written or quoted — every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.