Files
agent-estate-wiki/log.md
T

117 KiB
Raw Blame History

Wiki Log

Chronological record of wiki actions. Append-only.

[2026-07-22] init | Fresh Hermes Systems Wiki

  • Created technical wiki at /home/hermes/wiki
  • Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
  • Wrote SCHEMA.md with 11 operating rules
  • Wrote index.md as curated Start here dashboard
  • Reseeded system/infrastructure/runbook stubs for existing homelab
  • Git initialized, local-only
  • No secrets, passwords, tokens, or private keys written to vault

[2026-07-22] audit | External audit pass (Claude, at Anthony's request)

  • Merged duplicate pages: systems/omniRoute.md (empty stub) -> kept systems/omniroute.md (populated); repointed 5 links
  • Merged duplicate pages: runbooks/safely-update-hermes.md -> kept as runbooks/update-hermes-safely.md (fuller content); repointed 2 links
  • Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
  • Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
  • Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
  • Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
  • Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts

[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)

  • Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
  • Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
  • Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
  • Fixed a broken review_after: 0001-01-01 in decisions/keep-openrouter-as-fallback.md
  • New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
  • queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate

[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)

  • Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
  • Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
  • Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
  • Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
  • Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
  • Confirmed: Mnemosyne is, and has only ever been, this box's memory backend

[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)

  • Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
  • These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki

[2026-07-25] operations | Tool Search and gateway resilience

  • Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
  • Documented how new skill categories are derived from the first folder below ~/.hermes/skills/
  • Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
  • Recorded the I/O-controller delegation limitation so an ineffective IOWeight is not mistaken for active protection
  • Added health checks, exact effective settings, backup location, and rollback procedure
  • No secrets, tokens, or private credentials were written to the wiki

[2026-07-28] dashboard | Jobs Dashboard feature additions

  • Fixed bulk status update (payload shape mismatch: {ids, status} → {ids, updates: {status}})
  • Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
  • Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
  • Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
  • Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion Notes / JD Summary
  • Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
  • Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
  • Updated Synergy and Dept of Communities to Phone Screen status
  • Updated wiki runbook with full dashboard feature documentation and API endpoint table

[2026-07-27] incident | Job Radar search and dashboard recovery

  • Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
  • Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require MISSING_COVER_LETTER_COUNT=0
  • Recorded the production cron model (gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5)
  • Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
  • Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both :443 Serve and :8443/webhook Funnel
  • Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
  • Verified no secrets, API keys or private credentials were written to the wiki

[2026-07-29] update | Steel Browser deployment — browser backend switch

  • New page: systems/browser-backend.md — rewritten to reflect Steel Browser as primary, Camoufox as fallback
  • New page: comparisons/steel-vs-camoufox.md — comparison of the two backends
  • Updated: infrastructure/hosts.md — added CT201 and CT460 to verified hosts table
  • Updated: infrastructure/docker-services.md — added Steel Browser compose file
  • Updated: infrastructure/tailscale.md — added CT201 Steel serve routes
  • Updated: index.md — browser-backend link description
  • Updated: current-state.md — added 2026-07-29 changes section
  • Config change: browser.cdp_url set to http://100.96.244.39:9223 in ops profile

[2026-07-30] update | Open WebUI updated on CT245

  • Updated image to ghcr.io/open-webui/open-webui:main (new digest e97bf9531916, Jul 27)
  • Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
  • Restarted container after OOM hang
  • Updated infrastructure/proxmox.md

[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed

  • Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
  • Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
  • Ran mnemosyne reindex --yes to rebuild all embeddings — working memory now at 100%
  • Updated health check script (mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py)
  • Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
  • Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
  • DB grew from 8.7M to 21M with the new embeddings

[2026-08-01] incident | Mnemosyne capture pipeline repaired

  • Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no working_memory row existed.
  • Corrected stringified structured YAML (sync_roles, skip_contexts, reflect, sleep_threshold, ignore_patterns) and enabled profile isolation; hermes config check passed.
  • Verified the repair with a second CLI marker that persisted as an exact normalized [USER] database row.
  • Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route :9119; added authoritative identity/preference/workflow slots.
  • Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
  • Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.

[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation

[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync

  • Ops profile (~/.hermes/profiles/ops/config.yaml): enabled mnemosyne plugin with profile_isolation: true, allow_tool_override: false, shared_surface_read: false
  • Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
  • Verified end-to-end: memory writes land correctly in both profiles independently
  • Updated systems/mnemosyne — profile isolation section now documents ops config
  • Updated current-state — added ops profile Mnemosyne entry
  • Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
  • Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision

[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation

  • Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances
  • Reworked deepseek-flash-free and good-free on CT600: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort
  • Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
  • Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
  • Updated systems/omniroute — added Recent Changes (2026-08-04) section
  • Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated updated date

[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite

  • OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. is_active=0 does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
  • Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
  • Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
  • Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
  • Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
  • Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
  • Updated index.md — last-updated date 2026-08-04 → 2026-08-07

[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review

  • Portainer password reset documented: CT245 admin user confirmed as anthony (not admin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state.
  • SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint, // root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.

[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review

  • Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker from freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named jobs (e.g. vertex/gemini-2.5-flash, deepseek-v4-flash-free) → smart-route. Daily Marketing Job Radar → gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run (ID a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, CT333 Nanobot has live jobs.json with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
  • Updated current-state — added 2026-08-08 changes section
  • Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues

[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue

  • Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
    • TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
    • RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
  • Root cause identified: Disk at 83% (11GB free on 65GB), state.db at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
  • Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
  • Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
  • OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
  • No new pages created — all changes applied to existing pages

[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals

  • OmniRoute AutoRoute variants documented: auto/best-free (Auto Cheap) and auto/coding-free are virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: auto/best-free routed to Antigravity. Updated systems/omniroute.
  • OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains memory.provider: mnemosyne. Added to comparisons/memory-backends.
  • FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default freellmapi-auto. Added to comparisons/model-routers.
  • No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.

[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration

  • OmniRoute migrated from CT600 to CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list, grep, Tailscale DNS).
  • Updated systems/omniroute — Current Configuration rewritten for omni/CT601, added Recent Changes (2026-08-13), bumped updated/verified_on to 2026-08-13.
  • Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped updated.
  • Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
  • Updated infrastructure/hosts — OmniRoute proxy entry → omni; CT600 reference marked verified/retired.
  • Updated index — last-updated date 2026-08-10 → 2026-08-13.
  • No new pages created; all changes applied to existing pages. No secrets written.

[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated

  • Skills index drift incident: systems/skills-index.md claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating regenerate-skills-index.sh script, and adding cron job 1a8e90ea2271 (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift.
  • Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped updated/verified_on to 2026-08-15.
  • Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
  • Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
  • Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
  • Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
  • No secrets written.

[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash

  • New page: systems/nextcloud.md — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
  • Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
  • Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
  • Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
  • No secrets written.

[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk

  • OmniRoute upstream capacity pressure documented: ServiceUnavailableError: Structurally heavy chat request capacity is busy reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
  • Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
  • Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config http://omni:20128/v1 is correct — /v1 is required.
  • Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped updated/verified_on to 2026-08-16.
  • Updated current-state — added Recent Changes (2026-08-16) section, bumped updated to 2026-08-16.
  • Updated index.md — bumped last-updated date to 2026-08-16.
  • No new pages created. No secrets written.

[2026-08-18] incident | Mnemosyne capture pipeline broken (active)

  • New page: incidents/2026-08-18-mnemosyne-capture-pipeline — working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
  • Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped updated/verified_on to 2026-08-18, added cross-reference to Related.
  • Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped updated to 2026-08-18.
  • Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
  • Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from active to resolved (that earlier incident was fixed).
  • No secrets written.

[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history

  • Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
    • wiki-stocktake-maintenance (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from cx/gpt-5.6-luna → go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.
    • CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
    • Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
  • Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped updated/verified_on to 2026-08-23.
  • Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped updated/verified_on to 2026-08-23.
  • Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
  • Updated index.md — bumped last-updated date to 2026-08-23.
  • No secrets written.

[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status

  • New page: systems/ignis.md — documented Ignis browser-native Obsidian (Docker ignis-subpath:0.8.9, port 8080, Tailscale Serve at hermes.kangaroo-eel.ts.net/ignis). Volume mount: /home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active.
  • Updated systems/model-providers — added command-code provider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumped updated/verified_on to 2026-08-19.
  • Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider, deepseek-flash-mix combo, and Tavili search proxy status. Bumped updated/verified_on to 2026-08-19.
  • Updated infrastructure/docker-services — added Ignis compose file and service definition.
  • Updated infrastructure/hosts — added Ignis to CT460 row.
  • Updated systems/obsidian — added Ignis as browser interface, corrected vault path to agent-estate-wiki/, added Quartz static site reference.
  • Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
  • Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
  • No secrets written.

[2026-08-26] update | Nightly wiki audit — Cron job reconciliation

  • Updated systems/scheduled-tasks — reconciled live hermes cron list (27 jobs) against wiki table (29 entries). Removed: Workday start reminder (no longer in live list), CT 221 Miner Watchdog (job d5516d0a5147 removed; CT221 status unknown), Review Agent Estate Phase One (one-time job completed 2026-08-08). Added: weekly-save-restricted-bot-restart (0 3 * * 0, local, no-agent). Updated count from 29 to 27.
  • Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
  • Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
  • Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
  • No secrets written.

[2026-09-02] incident | SFTPGo outage on CT299

  • New page: incidents/2026-09-02-sftpgo-outage.md — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), tailscaled died; PVE pct start failed due to 5 stale lxc-attach processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop 299 --skiplock + pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
  • Updated current-state — added Recent Changes (2026-09-02) section, bumped updated/verified_on to 2026-09-04.
  • Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
  • No secrets written.

[2026-08-28] update | Department of Communities role and social-media audit

  • New page: reports/communities-wa-social-media-audit-2026 — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
  • Updated systems/current-profile — recorded current employment context and linked the report.
  • Updated index.md — added the report to the Reports section and refreshed the last-updated date.
  • No secrets written.

[2026-09-20] update | Nightly wiki audit — No changes in past 24h

  • No new pages created — no new systems, decisions, or incidents in the past 24 hours.
  • No existing pages updated — all wiki content remains current.
  • Freshness audit run: freshness-20260920.txt generated at 03:01 AWST (scheduled cron).
  • Skills index auto-regenerated: systems/skills-index.md updated at 03:01 AWST (scheduled cron 1a8e90ea2271).
  • Scheduled tasks reconciled: systems/scheduled-tasks.md updated at 03:02 AWST — 26 active jobs verified from live hermes cron list.
  • No secrets written.

[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation

  • New page incidents/2026-09-28-hermes-managed-runtime-deps — the 2026-09-28 → 30 cron-fleet outage: jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused by ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in the uv-managed runtime venvs under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (all 10 venvs, pip install --target), run every 15 min by job 96eb95ce12f2.
  • Updated systems/scheduled-tasks.md — table rebuilt from live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to d40085da631c, f77b7e46bd09, 0a7f933d06e2. Known Issues refreshed: job-radar 403 MODEL_NOT_IN_PLAN (streak 5), Telegram thread-delivery failures, weekly-save-restricted-bot-restart script restored, residual pre-fix dispatch errors on cbfe434a3b5e/1a8e90ea2271.
  • Updated systems/hermes-agent.md — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model custom → http://100.78.63.20:20128 / smart-route; skills 330 → 341; cron count 29 → 30.
  • Updated current-state.md — _config_version: 46 and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
  • Updated runbooks/job-radar.md — corrected primary model row (custom:omni-101 / gpt-5.6-terra via command-code) and added the 403 MODEL_NOT_IN_PLAN failure mode with recovery steps.
  • Updated systems/messaging-integrations.md — Telegram allowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
  • Updated index.md — last-updated to 2026-09-30, cron count to 30, new incident listed.
  • No secrets written.

[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene

  • No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
  • Updated systems/scheduled-tasks.md — table and counts re-verified against live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — dca8482e4f76 (403 MODEL_NOT_IN_PLAN, ×5), 1a8e90ea2271 (HTTP 410, model retired upstream, ×7), fad5a86979d4 (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: 1b23c042e17d and c8d52904e1d6 delivery targets now telegram:-1004321904721:102; c0a9fa314542 flagged for a revoked Telegram bot token; bare-telegram targets noted on 0e753e9720fd, fad5a86979d4, 066bb5191280. Known Issues restructured into Resolved / Open / Historical.
  • Updated systems/messaging-integrations.md — new Delivery target semantics section: a bare deliver: "telegram" resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails with Message thread not found and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
  • Updated systems/model-providers.md — provider stack rebuilt from live config.yaml: 5 custom providers (omni-101 5,384 models, Omniroute:20128 5,398, litellm 1,728, command-code 71, guacano 46); custom:omni recorded as removed. Default model corrected resilient-free → smart-route. New Retired / Unavailable Models table (retired qwen3.5:397b, plan-gated gpt-5.6-terra, catalog-missing deepseek-v4-flash) with the lesson that a raw model pin never self-recovers.
  • Updated systems/mnemosyne.md — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as [USER] [Anthony Martin] memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
  • Updated systems/hermes-agent.md — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
  • Updated index.md and current-state.md — last-updated bumped to 2026-10-01 with a Recent Changes block.
  • Vault sync: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under wiki/, not at the tree root, and index.md sits at the vault root.
  • No secrets written.

[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split

  • New page: incidents/2026-10-01-cliproxy-connection-auth-expired.md — job 06a0a1c8704f (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard (provider custom:Omniroute:20128, model cliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d, 79f9bb41ecf5).
  • Updated systems/scheduled-tasks.md — counts re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01: 06a0a1c8704f (HTTP 401), 0e753e9720fd (HTTP 400 — same NVIDIA model pin as the already-failing fad5a86979d4, confirming the shared-pin diagnosis), 066bb5191280 (delivery failed; report itself generated). Streaks advanced: dca8482e4f76 ×5→×6, 1a8e90ea2271 ×7→×8, fad5a86979d4 ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
  • Updated systems/messaging-integrations.md — corrected a wrong page fact. The page previously stated that a bad thread_id always loses the message. The 2026-10-01 records show two variants: Message thread not found (hard failure, message lost — the 2026-09-30 case) and configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id (a thread-less fallback send does occur, yet the run is still recorded delivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegram jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
  • Updated systems/current-profile.md — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no ~/.hermes/profiles/default/ directory (the default profile is the top-level config; ops is the only named sub-profile), the model default is smart-route (not freellmapi-auto), the default provider is custom → http://100.78.63.20:20128 / OmniRoute (litellm is now only the last fallback entry), and the config is _config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
  • Updated current-state.md — new Recent Changes (2026-10-02) block; frontmatter dates bumped.
  • Updated index.md — last-updated to 2026-10-02; cron line now notes the failing count.
  • Structural finding — the wiki exists as two trees and they have drifted both ways. /home/hermes/workspace/agent-estate-wiki is canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads (pages under wiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions — systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md and reports/communities-wa-social-media-audit-2026.md exist only in canonical — so canonical index.md currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), failing set still 6 but the composition moved. 1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard that the cliprox incident predicted — making it 2 of the 3 cliprox/terra jobs confirmed broken, which retires any "one-job fluke" reading. 79f9bb41ecf5 is the last untested job on that pin (next due 2026-10-04 18:00). Streak recount against executions.db: dca8482e4f76 ×6→×7, but 1a8e90ea2271 ×8→×7 — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). Resolved: 1b23c042e17d's stale delivery_failed flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). Downgraded: the GitHub Trending Radar revoked-token claim is now "unverified at last run" — c0a9fa314542's 2026-09-30 19:22 run completed and delivered. New Known Issue recording that a delivered notice is not a successful job (both cliprox failures read delivery_outcome: delivered on a failed status).
  • New page incidents/2026-08-26-work-outlook-basic-auth-disabled.md — the work Outlook mailbox has been unreachable since 2026-08-26: himalaya is configured for password auth while the tenant advertises only XOAuth2, so himalaya -a work dies with login not supported. TCP/TLS to outlook.office365.com:993 is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
  • Updated systems/messaging-integrations.md — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, work unreachable; frontmatter updated/verified_on bumped. Added the tooling consequence: email/mail-lookup resolves the account before searching, so work-mail requests hit this wall and must be reported as account unreachable, not "no results found".
  • Updated systems/hermes-agent.md — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to six jobs across three model faults plus one credential fault, and a second entry added for the stock-take job being down; the Email: connected bullet corrected.
  • Updated current-state.md and index.md — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
  • systems/skills-index.md regenerated 341 → 344 — including a new skill, fleet-wide-failure-triage (devops). This was done by hand: job 1a8e90ea2271 is still blocked on HTTP 410 (retired qwen3.5:397b) and cannot self-recover. wiki-maintain.sh itself is healthy and was run manually, also writing raw/stocktake/freshness-20261003.txt — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under wiki/; index.md sits at the vault root.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair

Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.

New pages

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md — mnemosyne_recall had been scoring keyword-only: fastembed was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 without raising an error. Repaired 2026-10-03; the runtime guard now asserts fastembed + FASTEMBED_CACHE_PATH + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
  • incidents/2026-10-03-ct450-hermes-update.md — CT450 (hermesagent) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (ssh root@100.96.100.82 → pct exec 450 -- …) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).

Updated

  • current-state.md — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the mnemosyne doctor / sqlite_vec misread gotcha.
  • systems/mnemosyne.md — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job b63ed5817212 documented.
  • systems/hermes-agent.md — version corrected to v0.21.5+6169.gab86a20 (was v0.21.5+3837.gbfda74c), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no profiles/default/; ops is the only named sub-profile), Known Issues bullet added.
  • systems/memory-backend.md — Known Issues bullet for the FTS-only degradation.
  • systems/scheduled-tasks.md — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), 79f9bb41ecf5 due 2026-10-04 18:00; bare-telegram delivery warnings confirmed live.
  • agents/hermes-production.md — version, Python, profile, provider routing, cron count and messaging refreshed; stale custom:litellm / freellmapi-auto references replaced.
  • agents/hermes-chaos.md — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
  • infrastructure/hosts.md — CT450 row updated to v0.21.5.
  • runbooks/update-hermes-safely.md — new "Second instance — CT450" section (access path + pre-flight checks).
  • runbooks/provider-health.md — stale v0.19.0 version reference corrected.
  • index.md — last-updated bumped; both new incidents linked; cron-fleet line re-dated.

Note — mnemosyne doctor on this host prints vec0: present_but_unloadable and marks the vector tiers unavailable. That is a Python-3.11-legacy-venv artefact, not the gateway's state. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py.

[2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7

Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json + hermes cron list: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 6 → 7. The one new failure event in the window is 79f9bb41ecf5 (Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. It was the last untested job on the cliprox/terra pin, so that pin is now 3 of 3 confirmed broken — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271) and the Known Issues Open list all updated; frontmatter dates bumped.
  • Updated incidents/2026-10-01-cliproxy-connection-auth-expired.md — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still Open — needs the OmniRoute connection re-authenticated before any repin.
  • Updated current-state.md — new Recent Changes (2026-10-05) block: the cliprox 3/3 settlement, 1a8e90ea2271's repeat 410 on 2026-10-04, confirmation that no interactive sessions occurred in the window (all 10 active sessions were cron), and a streak-counting caveat.
  • Updated index.md — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
  • Streak-number caveat recorded: hermes cron list's "N failures in a row" is the failure_streak counter in jobs.json, not a DB count. ~/.hermes/cron/executions.db now retains only rows from 2026-09-30 onward (1,001 rows total), so executions.db recounts undercount older streaks — the earlier DB-based "×7" for 1a8e90ea2271 undercounts against the live ×11. Quote the live counter and note the retention floor.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source — wiki/systems/scheduled-tasks.md, wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md, current-state.md, index.md, log.md.
  • No secrets written; all credential-shaped values remain referenced by connection ID only.

[2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0

Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.

New

  • incidents/2026-10-05-dead-model-pin-repin.md — the seven-job dead-model-pin repin. Records the four dead pins (cliprox/terra HTTP 401 ×3, ollama-cloud/qwen3.5:397b HTTP 410 ×1, nvidia/deepseek-ai/deepseek-v4-flash HTTP 400 ×2, command-code/gpt-5.6-terra HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (free-stack ×6, smart-route ×1 on custom:omni-101) over another raw model name.

Updated

  • systems/scheduled-tasks.md — Count block rewritten: 0 failing (was 7), with all 30 failure_streak counters at 0; four jobs carry a stale pre-repin last_status: error that clears on their next scheduled run. Seven job-table rows updated (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd, dca8482e4f76), plus c0a9fa314542. New ### Resolved 2026-10-05 subsection in Known Issues; five resolved bullets removed from ### Open, leaving bare-telegram targets, the two-wiki-tree drift, the work-mailbox block, the stale custom:omni key and the delivered-notice rule as the standing open set.
  • index.md — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.
  • current-state.md — new Recent Changes (2026-10-06) block; frontmatter dates bumped.

Findings worth the ink

  • dca8482e4f76 (Daily Marketing Job Radar) delivered to topic 1030 at 23:56 — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for 79f9bb41ecf5 (23:39) and 06a0a1c8704f (23:45).
  • GitHub Trending Radar's revoked-token diagnosis is dead. c0a9fa314542 ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No getMe → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright.
  • delivery_outcome: delivered still does not mean the job worked. Every pre-repin failure recorded delivered, because the failure notice is itself the deliverable. Judge on the execution status column.
  • A stale notice opened the review. Anthony was prompted by a 85edf6772bca notice that had sat in topic 346 since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.
  • Window tally (executions.db, 2026-10-05 00:00 → 2026-10-06 00:15): 185 completed, 3 failed (all pre-repin, 03:00–07:00), 2 unknown (aborted first attempts of 79f9bb41ecf5 and dca8482e4f76 at 23:33/23:48, superseded by their successful re-runs) and 1 running (this audit).

Vault mirror: the new incident page plus systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day

Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — Count block corrected: the stale pre-repin last_status: error set is now three (1b23c042e17d, fad5a86979d4, 0e753e9720fd), not four — 1a8e90ea2271 cleared its own when it ran at 03:00. Added a Re-verified 2026-10-07 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 0 failed / 1 running; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
  • index.md — last-updated → 2026-10-07; cron-fleet line re-dated (0 failing as of 2026-10-07, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
  • current-state.md — new Recent Changes (2026-10-07) block; frontmatter dates bumped.

Findings worth the ink

  • 1a8e90ea2271 ran itself at 03:00 and proved the repin. wiki-stocktake-maintenance completed 2026-10-06 03:00:31 → 03:03:32 on custom:omni-101 / free-stack — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired qwen3.5:397b). It regenerated systems/skills-index.md (344 → 346 skills), wrote raw/stocktake/freshness-20261006.txt and committed to Gitea as 33f4e95. Its four HTTP-410 incidents (6df957a05ec6, d284168d3663, cda67de5b3e3, e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. The daily manual wiki-maintain.sh run can stop.
  • Vault mirror drift found and partly repaired. systems/skills-index.md in /home/hermes/wiki/agent-estate-wiki/ was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
  • Window tally (executions.db, 2026-10-06 00:00 → 2026-10-07 00:00): 176 completed, 0 failed, 1 running (this audit) — 177 rows. 14 sessions, all cron; no interactive (non-cron) sessions, and no new incident opened.

Vault mirror: systems/scheduled-tasks.md, systems/skills-index.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings

Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — new Re-verified 2026-10-08 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 1 failed / 1 running; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New Resolved 2026-10-07 Known Issues section, plus two new Open items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.
  • index.md — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.
  • current-state.md — new Recent Changes (2026-10-08) block; frontmatter dates bumped.

Findings worth the ink

  • The incident machinery had its first real workout since 2026-09-30 — and it worked. 62619aa6bcb3 (DHL Tracking) failed its 08:00 run inside ensure_browser() with TimeoutError: timed out on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident 62619a_f485ccfd4a0d (failure_type: timeout) was opened 08:00:52, alerted 08:00:58 and auto-resolved 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed.
  • The tracked parcel has arrived. The 14:00 run reported waybill 8052838382 as Delivered, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed.
  • Stocktake ran automatically for the second consecutive day (1a8e90ea2271, 2026-10-07 03:00, commit 8c63ef2; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once.
  • Two real defects surfaced from the stocktake's own log. (1) Step 3 of wiki-maintain.sh — the Quartz rebuild — is skipped on every run: there is no quartz.config.ts and no npx, so public/ is never refreshed and the published site silently ages. (2) skills-index under-reports by one: 347 SKILL.md files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md (the other duplicated basename, hermes-api-vault, survives because its two copies sit in different categories). Both are now Open items in systems/scheduled-tasks.
  • Unchanged, still carrying a stale last_status: error: 1b23c042e17d, fad5a86979d4, 0e753e9720fd — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-telegram targets and the custom:omni stale-key references are also unchanged.

Vault mirror: systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

[2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open

Window audited: 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days.

Pages written

  • incidents/2026-10-09-cron-provider-timeout-storm.md — new.
  • incidents/2026-10-09-memory-recall-relevance.md — new.
  • systems/scheduled-tasks.md — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
  • systems/mnemosyne.md — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
  • current-state.md — Recent Changes (2026-10-09) block; frontmatter dates bumped.
  • index.md — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
  • systems/skills-index.md + raw/stocktake/freshness-20261008.txt — the stranded 2026-10-08 03:01 stocktake artifacts committed.

Findings worth the ink

  • The fleet had its worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight remain open. The watchdog (96eb95ce12f2) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, i.e. it reported the storm rather than suffering it.
  • The 2026-10-08 window was never audited — this job's own 23:59 run failed with the same timeout (incident cbfe43_fa81052baaf7) and committed nothing.
  • The stocktake has a two-day gap. 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no freshness-20261009.txt exists. 1a8e90ea2271 is the one job still at failure_streak: 2.
  • An instruction was reported done and was not. Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". It is still enabled (paused_at: null) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
  • Memory recall relevance — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is not in live config (memory.mnemosyne carries no recall-weight keys; no top-level mnemosyne: key).

Vault mirror: the touched pages mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

2026-10-10 — Stocktake (manual run, wiki-maintain.sh)

  • systems/skills-index.md regenerated — 346 skills / 70 categories (unchanged count; frontmatter updated/verified_on refreshed to 2026-10-10). Live on-disk count is 347 SKILL.md; the one-skill gap is the known de-dup of productivity/obsidian-git-sync/obsidian-git-sync/.
  • Freshness audit → raw/stocktake/freshness-20261010.txt: 62 stale (≥21d), 23 fresh, 7 no-date of 92 pages. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No freshness-20261009.txt exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands.
  • current-state.md — new 2026-10-10 section. The 2026-10-09 timeout storm window re-tallied from executions.db (2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163 completed, 15 failed — the prior audit recorded 162/15+1 running; the stray running row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).
  • DHL monitor re-probed, still enabled. 62619aa6bcb3 — enabled: True, paused_at: null, 0 8,11,14,17,20 * * *, last_status: ok. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. No cron mutation performed by this audit — retirement stays open item #1.
  • infrastructure/tailscale.md verified against tailscale serve status — all 13 tailnet Serve routes plus the :8443/webhook Funnel match the page. No drift; not edited.
  • Not verified / left alone: the Quartz rebuild step (no quartz.config.ts, npx cannot fetch quartz — public/ was NOT refreshed, confirming the carried open item); docker ps (permission denied on /var/run/docker.sock from this session, so infrastructure/docker-services.md could not be probed and is left stale); the 80d 2026-07-22 cluster (no live source to check against).
  • Sessions in the 2026-10-09 window (state.db, epoch REAL): 19 total — 6 cron, 11 ios, 1 talaria-ios, 1 telegram. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the delivered count includes failure notices).

No secrets written; job/incident IDs only.

[2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-11T00:19). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 174 completed / 0 failed. Non-clean last_status 10 -> 5. Incidents: 7 closed during 10-10, 5 open (3 jobs). Open item #1 closed: the DHL monitor 62619aa6bcb3 is retired and verified absent from jobs.json by ID lookup (the 10-07 and 10-09 "deleted" claims were false).

Runtime/memory. The 2026-09-28 managed-runtime-dependency class recurred in the gateway env: mnemosyne (+native deps) and elevenlabs were missing from the committed uv env 795020dbf24343cd9c52df24723dca83 that activate_dependencies() loads. Repaired 21:26-21:34 and verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.

New page. systems/paseo - Paseo daemon v0.11.2 on 0.0.0.0:6767 (user unit paseo-daemon.service); recorded the daemon run vs daemon start supervisor trap, the /usr/bin/paseo 0.7.2 vs daemon 0.11.2 version split, and the truncated-cline false provider fault.

Platform. Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 answers neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.

Pages touched: systems/paseo.md (new), systems/scheduled-tasks.md, systems/hermes-agent.md, systems/mnemosyne.md, systems/messaging-integrations.md, incidents/2026-09-28-hermes-managed-runtime-deps.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-11] stocktake | Daily maintenance + live drift corrections

  • Ran wiki-maintain.sh: skills-index regenerated (346 skills / 70 categories), freshness audit written to raw/stocktake/freshness-20261011.txt (62 stale / 24 fresh / 7 no-date of 93 pages)
  • OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed. Verified via pct list, pct config, ss and tailscale status: CT600 omniroute (100.88.81.19) is RUNNING and serving :20128; CT601 omni (100.93.204.4) is STOPPED with onboot=0. Hermes resolves custom:omni → omniroute:20128 → CT600, not CT601. systems/omniroute.md rewritten with the live node table.
  • Third OmniRoute node found: omniroute-1 at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider omni-101 (5384 models, default smart-route, carries skills_hub). Not in pct list — identity unconfirmed.
  • Ignis duplicate found: a second ignis-ignis-1 container runs on CT200 (image ignis-server:latest, host port 8092, vault /opt/ignis/vaults), separate from CT460's ignis (port 8080, vault /home/hermes/wiki/agent-estate-wiki). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them.
  • Tailscale route table corrected against live tailscale serve status: /agent-estate target is bare http://127.0.0.1:8769 (not .../8769/agent-estate-role-dashboard.html); previously-missing tailnet-only :9443 → 127.0.0.1:5173 documented. CT201 routes refreshed (8085 targets + /steel 5173 + /steel-api 3000).
  • Proxmox inventory rebuilt from live pct list + qm list: added CT475 agentzero and VM110 omarchy; corrected 282 ubuntu-server-xfce to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). VM403 open-claw and VM401 hermes-old are absent from qm list — flagged unconfirmed rather than guessed.
  • Nanobot verified on CT333: gateway nanobot-gateway (PID 2259048) listening on 8765/8766; NOT a systemd nanobot.service/gateway.service (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified.
  • Docker services page corrected: the old "Docker runtime not available" note was a CT460 socket-permission artifact (/var/run/docker.sock is root:docker srw-rw----, hermes user not in group docker, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo).
  • Flagged for follow-up, root cause not investigated: mycompose-db-1 on CT200 is in Restarting (1) — crash-looping.
  • Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
  • Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.

[2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-12T00:10). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 175 completed / 0 failed (finalised), 10-11 171 completed / 0 failed + 1 running — two consecutive clean days. Non-clean last_status steady at 5; 0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected).

Corrected a false alarm instead of logging a fault. The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a naive-UTC vs AWST comparison error — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in systems/mnemosyne and systems/scheduled-tasks so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.

New content. queries/retained-research-answers was a 2026-07-22 stub and is now a real page — the 2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; -CompressLevel 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage was undocumented; the live datastore table plus host mount points are now in infrastructure/proxmox (/mnt/ssd0 93 % / ~63 GiB free is the pressure point).

Stale labels fixed. incidents/2026-08-18-mnemosyne-capture-pipeline carried status: active from August despite capture being verifiably healthy — now superseded, with the live evidence and a caution about the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open, and the skills-index blurb no longer claims no regeneration since 2026-10-07.

Verified, not changed. The blocked-page-recovery lesson that session reported patching (recovered: false = archive routes failed, not task failed; API-first rung) is present in the live skill.

Pages touched: systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/proxmox.md, queries/retained-research-answers.md, incidents/2026-08-18-mnemosyne-capture-pipeline.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections

Script run. wiki-maintain.sh completed all three stages: skills-index regenerated from the live skill tree (346 skills across 70 categories, unchanged count), freshness audit written to raw/stocktake/freshness-20261012.txt (54 stale / 32 fresh / 7 no-date of 93 pages), and the Quartz step took its documented no-quartz.config.ts path (build skipped, public/ unchanged — expected, not a failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and written its artifacts, so results were read from the report rather than assumed.

Drift corrected from live probes (2 pages):

  1. systems/nanobot — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on CT333 is back up, HTTP 200 on http://100.122.189.81:8766/. Port 8900 — the endpoint the disabled OmniRoute nanobot provider connection uses — is still closed, so status stays degraded rather than flipping to active. SSH to CT333 is unreachable from CT460, so whatsapp-bridge.service was not re-verified.
  2. systems/home-assistant — carried confidence: low plus a page-wide "cannot confirm HA runs" caveat from 2026-07-22. Live: HTTP 200 on http://100.68.139.5:8123/, Tailscale node homeassistant active and offering an exit node, VMID 100 running per qm list. Confidence raised low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).

Verified and left stale. systems/openrouter and systems/nextcloud were probed shallowly and nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.

Pages touched: systems/nanobot.md, systems/home-assistant.md, systems/skills-index.md, raw/stocktake/freshness-20261012.txt, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes

Window. 2026-10-12 00:00 → 23:59 AWST (sessions from ~/.hermes/state.db, cron from ~/.hermes/cron/executions.db + cron_incidents; live probes 2026-10-13 00:05–00:45). One former compaction of this job left part of the review written but the log entry and cross-page updates unfinished — completed here.

Headline 1 — a clean run hid a dead credential. c0a9fa314542 (GitHub Trending Radar — Weekly Roundup, Mondays 09:00) reported completed with delivery_outcome: delivered on 2026-10-12, but its fetch-then-send script died at digest.py send(). Live probe of the bot token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 from getMe — the token is a valid 46-character shape, i.e. revoked, not malformed. The fetch path is healthy (100 repos, 26 cache files in 24 h). Critically, this supersedes the 2026-10-06 entry on current-state, which concluded the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in systems/scheduled-tasks: a completed run does not clear a credential fault unless the run exercised the same call that failed, and green cron status is not delivery proof. (Credential was present; value neither recorded nor repeated.)

Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new). The 2026-10-12 07:01 Daily Marketing Job Radar run drove camofox-browser.service to 2.5G, swap fully consumed, available: 0B, 383 tasks against the drop-in ceiling MemoryHigh=2G / MemoryMax=3G / MemorySwapMax=1G. REST /tabs and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at 07:23:50, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis: workload exceeded the cgroup, not a leak — OOMPolicy=kill never fired.

Headline 3 — systems/browser-backend corrected against live probes. Prior revisions were wrong on three counts: the REST API listens on :9377 only (:9093 closed, :9378 does not exist — REST and WebSocket are the same service, not two), Camofox runs as a systemd --user unit on the Hermes host (CT460), not CT450, and the containment drop-in was undocumented. current-state.md also still described the old port model; both pages now match the probe.

Cron state. 2026-10-12: 174 completed / 0 failed / 1 running (this audit, cbfe434a3b5e, started 23:59:41) — third consecutive clean day; 2026-10-11 finalised at 172 / 0. Retention floor moved to 2026-10-07 08:15. Fleet 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3 (0e753e9720fd error, 1b23c042e17d error, 066bb5191280 delivery_failed). fad5a86979d4 (Facebook Birthday Weekly Registry Sync) and dca8482e4f76 (Daily Marketing Job Radar) both cleared on 2026-10-12 — the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.

Incidents — open 5 → 2. Closed on 2026-10-12: fad5a8_a13bbc837e1c + fad5a8_c80b59a907f0 at 03:05:39 (the two deepseek-v4-flash HTTP-400 rows, cleared by the registry sync's clean run) and dca848_23fcd2b18aaa at 07:33:41. Remaining: 1b23c0_c35703f637b8 (alerted — weekly job, cannot clear before Fri 2026-10-16 08:00) and 0e753e_e7f8194a7d06 (detected since 2026-10-01 — the job has no retained executions, so it cannot clear until 2026-11-01). → incidents/2026-10-09-cron-provider-timeout-storm updated from "8 still open" to 11 of 12 closed with the full closure timeline; the pattern recorded there is that each incident clears only on the next successful run of its own job, so weekly jobs carry their rows for up to a week after the fault is gone.

Mnemosyne (2026-10-12 12:02 health run, green). Working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 % — both above the 95 % floor, no reindex; orphan embeddings 0; FTS5 exactly aligned; 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → systems/mnemosyne.

Sessions. 18 in the window, all cron, zero interactive. The 03:00 stocktake 1a8e90ea2271 ran clean and pushed 0eee5f1 (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 categories — the known 347-vs-346 on-disk off-by-one persists).

Pages touched: current-state.md (new Recent Changes section + superseded-correction tag), systems/scheduled-tasks.md (frontmatter, job inventory, non-clean statuses, re-verified history, five job rows, ### Resolved 2026-10-12, corrected incident IDs, ### Open additions, new ## Re-verified 2026-10-13 block), systems/browser-backend.md (port + host + containment rewrite, history entries), systems/mnemosyne.md (12:02 coverage), incidents/2026-10-09-cron-provider-timeout-storm.md (status + full closure table + timeline), index.md (date, four blurbs), log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential

Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.

Cron. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight → the day finalises at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 — the first failed run since 2026-10-09 (which recorded 15). Retention floor moved to 2026-10-08 07:11 (MAX_TERMINAL_EXECUTIONS = 1000; 1,001 rows = 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads 126 / 1, partial).

Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent, alerted_at: NULL — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three on the opencode path (10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode; 10-09 timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00 run. → incidents/2026-10-05-dead-model-pin-repin

Finding 2 — the failure stranded its own generated artifacts (now committed). Step 1 (skills-index) and step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working tree: systems/skills-index.md (modified, date-only: updated/verified_on 10-12 → 10-13; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked; 48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit. Consequence: the stocktake published nothing on 10-13 — and because the agent row is what commits, even a perfect script run leaves nothing behind when the LLM turn dies.

Delivery. 159 suppressed / 9 delivered / 1 no-record. Delivered: Daily Marketing Job Radar, Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday, Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 2026-10-01.

Mnemosyne. Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no reindex was needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than bare; the delivery layer still suppressed it (delivery_outcome: suppressed) — the matcher tolerates wrapping rather than requiring the literal token as a strict control. → systems/mnemosyne

Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive session.

Fleet. 29 jobs (26 enabled / 3 disabled), pins unchanged; non-clean last_status 3 → 4 (cleared fad5a86979d4 10-12 03:05 and dca8482e4f76 10-13 07:02; added 1a8e90ea2271); open incidents 2 → 3 (1 alerted, 2 detected, 8 closed, 66 resolved; 77 rows).

Pages touched: systems/scheduled-tasks.md (frontmatter, retention floor, non-clean list, ## Re-verified 2026-10-14), systems/skills-index.md (committed stocktake artifact), current-state.md (new Recent Changes + date), index.md (date + two blurbs), log.md.

Vault mirror. The 2026-10-13 audit was never mirrored to /home/hermes/wiki (last mirror bb09581, 2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus skills-index.md, whose vault copy was still the 2026-10-08 version. The vault's own raw/stocktake/ still holds only the 2026-10-03 report.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt

Ran wiki-maintain.sh (03:01:09 → 03:01:10 AWST): step 1 regenerated systems/skills-index.md (346 skills / 70 categories), step 2 wrote raw/stocktake/freshness-20261014.txt (48 stale / 38 fresh / 7 no-date of 93 pages), step 3 skipped the Quartz rebuild again (no quartz.config.ts, no npx) so public/ is untouched — the known carried item.

Finding 1 — a second, unclustered Proxmox hypervisor exists: m93p (100.104.51.98), PVE 9.2.0 (kernel 7.0.14-12-pve). pvecm nodes is empty on both pve and m93p — no cluster — and guest IDs collide between them (m93p CT221 is gitea; pve CT103 is cliproxyapi; CT100 and CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the hypervisor port :8006 returned only m93p and pve. m93p had been sitting in the Tailscale node list for weeks as an unremarkable online peer, with no wiki page recording it as a hypervisor. Live guests recorded from a single pct list / qm list pass.

Finding 2 — four guests the wiki placed on pve live on m93p. CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are absent from live pct list on pve, yet every one of the four services answers. /etc/pve/lxc/ on pve contains only 260.conf.bak and no vzdump task log references CT 221/241/245/260 — so they were not deleted, they were never on this host. The 2026-10-11 inventory was wrong and every audit since carried it forward.

Finding 3 — the tailnet inventory had drifted badly. The page listed 30 nodes; live tailscale status reports 45 non-funnel nodes (plus 23 funnel-ingress-node IPv6 peers, now noted and deliberately excluded). Fifteen nodes were missing, including m93p, pve, omniroute-1, paseo, qwenpaw, rustdesk, vaultwarden, social-dashboard, openweb, omarchybook, sftpgo, rclone, omni and runtipi. Several states were wrong too (anthonys-mac-mini is online on a different IP; omarchy-macmini / omarchy-quattro are not live peers; anthonys-iphone is online, not active). The CT460 serve routes re-verified exactly as recorded — 13 routes plus :8443/webhook Funnel and :9443 tailnet-only — so that table needed no change.

Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling: camofox-browser.service now reports Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M) with 16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the fault, as diagnosed last cycle.

Also verified, unchanged: Mattermost CT255 stopped (peer offline 24d); VM 282 ubuntu-server-xfce is now running, not stopped as recorded.

Cron (verified from executions.db / jobs.json, 03:05): 2026-10-13 = 168 completed / 1 failed (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress 23 completed / 0 failed with the 03:00 stocktake 1a8e90ea2271 still running at probe time. Open incidents 3 — 1a8e90_d11698bd7275 (detected, 10-13 opencode 404), 0e753e_e7f8194a7d06 (detected since 10-01, next slot 2026-11-01), 1b23c0_c35703f637b8 (alerted since 10-09, next slot Fri 2026-10-16 08:00). Non-clean last_status: 4 (unchanged). Fleet 29 jobs — 26 enabled / 3 disabled.

Pages touched: infrastructure/proxmox.md (two-host rewrite, m93p inventory, four-entry correction, storage, VM 282), infrastructure/tailscale.md (45-node inventory + funnel-peer exclusion note, frontmatter), infrastructure/network-map.md (m93p entry, frontmatter), current-state.md (new Recent Changes block), index.md (proxmox/tailscale blurbs + header warning), systems/skills-index.md (stocktake artifact), raw/stocktake/freshness-20261014.txt (new), log.md.

Not verified, left as stale: the remaining 46 stale pages were not probed against a live source, so they stay flagged rather than rewritten. Not re-verified this pass and still single-host in framing: systems/omniroute, infrastructure/hosts, systems/model-providers, infrastructure/docker-services — m93p also runs its own omniroute (CT101), so the OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.


[2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node

Ran 2026-10-16 00:0x–00:3x AWST. Scope is two windows, not one: the 2026-10-14 23:59 Wiki Audit (cbfe434a3b5e) failed on the fault below, so the 10-14 window was never covered.

Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).

  • Root cause. httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101 (confirmed in config.yaml: custom provider name: omni-101, base_url: http://100.78.63.20:20128). It surfaced as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. — the exact string recorded on every failing execution row. 328 Errno 111 lines in the live errors.log, 504 across the rotations.
  • Impact. 10-14 = 160 completed / 12 failed; 10-15 = 151 completed / 24 failed (+1 in flight). Against 10-12 175/0, 10-13 167/1. 36 failure rows across 13 jobs, but only 24 are genuine model-call failures on 12 jobs — the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
  • The finding that matters most — there is no failover. Verified from live jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable pin, custom:omni (CT600, omniroute:20128), sits on e530c89f2be0 — which is no_agent: true; d40085da631c is disabled. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni. So zero model calls completed in 34 hours, and the fleet has no job that could even report the second node's health.
  • Correction made during this audit. A first pass read Evening Stand-Down Briefing's clean 18:45 runs (both days) as evidence CT600 stayed healthy. Wrong: it is no_agent and never calls a model. Recorded as a standing rule — never read a no_agent job's green row as provider-health evidence — in the incident page and in systems/scheduled-tasks.
  • User-visible cost: silence. Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The weekly Perth New Restaurant was missed outright (next slot 10-22).
  • Recovery, verified live (not assumed). 1b31187397a5 completed 10-15 22:01:56 (streak 4 → 0); 96eb95ce12f2 10-16 00:01:36; b63ed5817212 10-16 00:02:29; ea79a2107e36 10-16 00:09:48. Probes 2026-10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19.
  • Fleet bookkeeping. Non-clean last_status: 3 → 12 (9 new: 08eb86f02d37, ecc1fc13655a, dca8482e4f76, c8d52904e1d6, 06a0a1c8704f, 85edf6772bca, 0e519874d280, 1a8e90ea2271, cbfe434a3b5e; three cleared in the window: b63ed5817212, 1b31187397a5, ea79a2107e36). Open incidents 3 → 12 (9 alerted / 3 detected). Long-standing and pre-storm: 1b23c042e17d (Fri 08:00), 0e753e9720fd (1st 09:00), 066bb5191280 (delivery_failed, dead Telegram thread 34795).
  • Mnemosyne. Both monitors sit on the failing node, so no coverage figures were refreshed for ~34 h; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected.
  • Self-healing gap left alone deliberately. The 10-15 03:00 stocktake failure left raw/stocktake/freshness-20261015.txt unwritten and systems/skills-index.md stamped 10-14. Its next slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the scheduled run.
  • Sessions. 36 sessions across the two days, all cron, zero interactive — fourth consecutive window with no interactive session.

Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new, 9,303 B); systems/scheduled-tasks.md (frontmatter, Count current-state bullet, new Re-verified block); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + blast-radius note on the two monitors); current-state.md (frontmatter + new top section, Updated: line); index.md (Last updated, new incidents entry, scheduled-tasks bullet); log.md (this entry). Nothing removed or archived — no retired job, no obsolete system, so index.md structure gains one page.

Durable fix still open (needs Anthony's call): the single-node topology. All 18 model-calling jobs on one OmniRoute node means this recurs on the next omniroute-1 blip — as it did on 2026-10-09. Options are a second provider pin on a subset of jobs, or router-side failover. Not applied — that is a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.

[2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead

Window: cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran 2026-10-18 00:31 → 00:5x AWST. Sessions reviewed: 30, all cron, zero interactive — fifth consecutive window with no interactive session, so every finding below is from live system state, not conversation.

Summary

  • omniroute-1 (custom:omni-101) has a second failure mode: 503 load-shedding. 9 model calls across 7 jobs died on RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. between 10-16 08:00:37 and 10-17 03:00:28 (~19 h). The control plane stayed up the whole time (/api/health 200 {"status":"ok"}, /v1/models 401 — re-probed 10-18 00:33), so a green health endpoint is not evidence that the model path is serving. This is not a repeat of the 10-14 storm: that was reachability (Connection error., Errno 111), this was the router serving and refusing under load. → incidents/2026-10-16-omniroute-503-resource-pressure (new).
  • The burst ended on its own and the fleet is green again. 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01), 10-18 so far 7 / 0; no failure row of any kind since 10-17 03:00. Every affected job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31, ecc1fc13655a + ea79a2107e36 both 10-17 12:02, each auto-closing its own incident. No change was made on the node; nothing was restarted or repinned.
  • Mosquito Tracker removed — fleet 29 → 28 (1b31187397a5). Absent from jobs.json (updated_at 2026-10-18T00:36:17+08:00), last execution 10-16 15:00 (completed, delivered). The three disabled jobs are unchanged, so 25 enabled / 3 disabled. Non-clean last_status 12 → 8; only 1b23c042e17d (error, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
  • The watchdog double-counts. 96eb95ce12f2 exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one sentinel. Read failures per job, not per row.
  • Second consecutive stocktake hole, again left alone deliberately. The 10-17 03:00 slot failed, so raw/stocktake/freshness-20261017.txt is missing and systems/skills-index.md is stamped 10-16; 10-15 is missing too (storm). The 10-18 03:00 slot closes it — no manual regeneration, because a hand-run would race the scheduled one.
  • GitHub Trending Radar is dead at the delivery end. The Telegram bot token in its config.json returns HTTP 401 on getMe (probe 10-18 00:35); the job's GitHub PAT still returns 200, so only the delivery credential is gone. The cron row stays green (last_status from the 10-12 09:00 run, last_run_at 2026-10-12T09:00:12). Next exercise Mon 2026-10-19 09:00. Not repaired — a new bot token is Anthony's call, and the secret is not written here.
  • Incidents: 9 opened, 6 closed in-window; open 12 → 8 (5 alerted / 3 detected). The ledger files 7 of the 9 503 rows failure_type: unknown despite the explicit message (the two script-exit rows are typed script), so failure_type still cannot be used to triage.
  • Mnemosyne: both monitors rode the burst (ea79a2107e36 10-17 00:00, b63ed5817212 10-16 12:03), both clean on every slot since — including 10-18 00:00 (delivery_outcome: suppressed). No fresh coverage figures exist (they report to Telegram, not the ledger); last explicit read stays 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %), above the 95 % reindex threshold, so no reindex is due.

Pages touched: incidents/2026-10-16-omniroute-503-resource-pressure.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + stocktake row + Known Issues entry); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + new Re-verified section); current-state.md (frontmatter + Updated: line + new top section); index.md (Last updated, new incident entry, scheduled-tasks bullet, skills-index note); log.md (this entry). Nothing removed or archived — no page became obsolete in the window.

Bridge housekeeping (verified 2026-10-18): every page this run touched was byte-identical between the authoring tree (/home/hermes/workspace/agent-estate-wiki) and the vault copy (/home/hermes/wiki/agent-estate-wiki, pages nested under wiki/) before the edits, so the seven files were written to the authoring tree first, committed and pushed (4137dd6), then copied into the vault — sha256 matched on all seven. The vault-only pages (wiki/systems/ignis.md, wiki/systems/nextcloud.md) and the vault's own wiki/raw/stocktake/ series were not touched; nothing was deleted and no one-way sync was run. Drift found, not fixed: incidents/2026-09-02-sftpgo-outage.md exists only in the authoring tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.

Durable fix still open (needs Anthony's call): the single-node topology — all 18 model-calling jobs on custom:omni-101 with no failover, now seen failing two different ways in five days. Options remain a second provider pin on a subset of jobs, or router-side failover. Not applied — a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.

[2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST

Window verdict: one failure all day, and it was a failure mode the wiki had never recorded. Fleet 28 jobs (25 enabled / 3 disabled); 10-18 = 168 completed / 1 failed against 10-17 = 163 / 5; 10-19 so far 5 / 0 plus this audit. 11 sessions in the window, all cron — no interactive sessions, so no user decisions arrived to document; every finding below is machine-observed.

  • A model stream repeated itself and Hermes delivered the placeholder to Telegram. ecc1fc13655a (FreshRSS Midday, 12:00) wrote "Your request was not processed. Send it again if you still want me to carry it out.", delivered it (delivery_outcome: delivered, 12:08:11) and then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response. New page → incidents/2026-10-18-degenerate-stream-canned-substitute. The open question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue) instead of substituting text on the job's normal route?
  • Mattermost: cause found — CT255 is stopped, superseding the 10-11 entry's "cause not determined". Reconnect loop unchanged (attempt 471 at 00:14:07, connect timed out after 30s, 5-minute cadence), peer offline 29d, and the host answers neither TCP :8065 nor ICMP. Platform config still says enabled: true, so the loop will keep logging until the container is started or the block retired.
  • New page: systems/push-notifications — the bridge is broken and has been for as long as the logs go back. status.json: connected: false, error: …connection refused (nothing listens there on this machine); the bridge dials HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net, which /etc/hosts resolves to 127.0.1.1 where nothing serves 443, while the gateway itself is healthy on 0.0.0.0:9119 (/ → 302) and Serve still maps the name to it. 23,469 occurrences of that string across agent.log(.1-3) + errors.log(.1-2). Fix documented, not applied.
  • Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break. working_memory had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18 03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in the page); 1 unresolved conflict. The 24 h threshold in mnemosyne-health-check.sh:92 will keep flagging idle weekends.
  • Stocktake landed and closed the two-day hole: 1a8e90ea2271 exited 0 on 10-18 03:00, freshness-20261018.txt written, systems/skills-index regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (dd53e5f).
  • Carried forward, still open: the GitHub Trending Radar delivery token (getMe → 401, verified 10-18) — its next exercise is Mon 2026-10-19 09:00; the single-node custom:omni-101 topology with no model-call failover; and the vault drift below.

Pages touched: incidents/2026-10-18-degenerate-stream-canned-substitute.md (new); systems/push-notifications.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + FreshRSS row + totals); systems/messaging-integrations.md (frontmatter + new Mattermost section); systems/mnemosyne.md (frontmatter + new tier/health block); infrastructure/tailscale.md (frontmatter + inventory header + three missing peers + mattermost age); infrastructure/proxmox.md (CT255 age); current-state.md; index.md; log.md. Nothing removed or archived — no page became obsolete and no one-way sync was run.

Bridge housekeeping + vault drift (verified 2026-10-19 00:2x). Of the ten files this run touched, six were byte-identical between the authoring tree and the vault copy before the edits; two more (infrastructure/tailscale.md, infrastructure/proxmox.md) differed only because the vault holds older versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (wiki/ nesting) with sha256 verified per file.

Vault drift measured: against authoring HEAD, the vault holds 74 identical pages, 11 stale and 12 missing. Stale: infrastructure/network-map.md, infrastructure/tailscale.md, infrastructure/proxmox.md, infrastructure/docker-services.md, systems/skills-index.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, incidents/2026-08-01-mnemosyne-capture-failure.md. Missing: incidents/2026-09-02-sftpgo-outage.md (flagged by the 10-18 audit and still absent), the seven agents/* pages, reports/communities-wa-social-media-audit-2026.md, CONTRIBUTING.md, plus this run's two new pages (copied immediately after). The vault repo's own HEAD is still 9bacd8e mirror(2026-10-16) with a working tree carrying uncommitted 10-17/10-18 changes, and it has no remote, so it cannot be pushed. Not fixed — a full mirror pass is a bigger call than an audit's mandate; flagged here.

No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.

[2026-10-20] audit | Nightly wiki audit — omni-101 503 burst 2 (2026-10-19)

  • Incident re-opened. incidents/2026-10-16-omniroute-503-resource-pressure went back to status: active. A second, identical HTTP 503 … resource pressure burst hit omniroute-1 on 2026-10-19 — 9 model calls across 8 jobs (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 (2026-10-16 08:00 → 10-17 03:00). Whole day 159 completed / 13 failed; 2026-10-18 was 168 / 1.
  • Facts corrected from the source of truth (~/.hermes/cron/jobs.json, executions.db) instead of being carried forward: the burst-2 table's job names, schedules and model aliases were wrong (85edf6772bca is Tech & AI Newsletter Digest (Analytical), not "Fringe Festival Plan"; ea79a2107e36 is the Mnemosyne Health Monitor, not a news digest; ecc1fc13655a is FreshRSS Midday News Briefing), 08eb86f02d37 fails at 08:31 (not 08:01, it is a 30 8 * * * job), and 2026-10-17 reads 163 / 5, not 162 / 5. All eight failing jobs read provider: custom:omni-101, model: smart-route — the burst-1 note's wider alias mix (deepseek-paid-smart, free-IA, dark) no longer matches jobs.json and is now flagged in the page for confirmation rather than silently overwritten.
  • Incident ledger read properly for the first time. Unclosed rows = 13 (12 alerted / 1 detected), not the 7 the 10-19 entry recorded — the earlier figures used a narrower filter. A raw closed_at IS NULL count reads 12 at 10-19 00:20 and 13 now; the entry states the query so future entries compare like with like. 4 rows opened and 2 closed on 10-19, plus ea79a2_747b27556fd0 closed 10-20 00:04:24.
  • Pages updated: incidents/2026-10-16-omniroute-503-resource-pressure.md (burst-2 table rebuilt from jobs.json, new findings, status back to active), systems/scheduled-tasks.md (new current-count bullet, non-clean last_status rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected), systems/omniroute.md (new Recent Changes section), current-state.md (new Recent Changes section), index.md (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), log.md.
  • Nothing removed or archived — no page became obsolete and no one-way sync was run.
  • Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 15 sessions in the window, all cron — no interactive sessions, so there were no user-facing decisions to record.
  • No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
  • Vault drift (measured — sha256 per file against /home/hermes/wiki/agent-estate-wiki): 76 identical / 15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512 "missing", but those are build artefacts under public/, package*.json, serve.py and raw/stocktake/, which the vault deliberately does not carry.) All six pages touched by this audit were copied in with per-file sha256 verification. The stale pages left unmirrored — infrastructure/docker-services.md, infrastructure/network-map.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, systems/skills-index.md, incidents/2026-08-01-mnemosyne-capture-failure.md — are authoring-tree edits made from other agents' sessions; a full vault mirror pass remains the open call.
  • Not re-run this audit: the public/ Quartz rebuild and wiki-maintain.sh. Authoring commits 95fd7d3 → 766882e → 8da6ddf → 0248698, pushed to Gitea Anthony/agent-estate-wiki and confirmed at the remote.

2026-10-21 — nightly wiki audit (cron cbfe434a3b5e, window 2026-10-20)

  • A clean day — and a failure this ledger cannot see. 2026-10-20 closed 168 completed / 0 failed (plus 1 unknown), the first zero-failure day since 2026-10-12, ending a seven-day run. The omni-101 503 fault did not recur (last failure row 2026-10-19), so incidents/2026-10-16-omniroute-503-resource-pressure returns to monitoring. Zero incident rows opened, 4 closed → unclosed 13 → 9 (8 alerted / 1 detected).
  • New page: incidents/2026-10-20-gateway-unclean-exit-orphaned-run. The gateway died with no exit path at 2026-10-20 09:09:01 (last heartbeat 08:50:28, pid 2496376, suspected_oom=False), orphaning the 08:30 Gmail Action Digest run (08eb86f02d37, claimed 08:31:03) as unknown — "owner exited before a durable terminal state". No delivery row for the slot, no incident row created, and the job table still shows the previous slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits since 2026-08-01, 5 of them clustered on 2026-08-14/15).
  • New page: incidents/2026-08-08-gitea-config-backup-push-dead. e27da6c34bbb reads ok and its executions read completed nightly while nothing has reached Gitea since 2026-08-07 11:00:47 — 75 unpushed commits, fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'. Second instance of the "green row, dead job" pattern after the GitHub Trending Radar token revocation.
  • Corrected the 2026-10-18 canned-substitute page. Its "second surface" section attributed the interactive-session substitution to an HTTP 404 Model-not-found; the retained log evidence is Rebuilt-message restart limit (3) exceeded → reason=rebuilt_restart_limit_exceeded … api_calls=1/150 tool_turns=0 response_len=0. The page now carries the verified trigger and flags the unresolved model disagreement (session pin minimax/minimax-m3.1-flash:free vs model=qwen3.5:397b in the turn-end line). The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered, and is recorded as outstanding in current-state.
  • Pages updated: current-state (new Recent Changes section), systems/scheduled-tasks (new count bullet — non-clean last_status 12 → 9, config-backup row annotated, frontmatter), systems/current-profile (model default moved to nous / meituan/longcat-2.5-preview:free, no fallbacks key, pins unchanged), infrastructure/backups (the scheduled Gitea push has been dead since 2026-08-08), incidents/2026-10-16-omniroute-503-resource-pressure (status back to monitoring + dated update), incidents/2026-10-18-degenerate-stream-canned-substitute (trigger corrected), index.md, this log.
  • Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 20 of 28 pin OmniRoute; the 8 unpinned jobs moved upstream to nous with the global default change — the fleet now runs two upstream paths. 2 interactive sessions in the window (talaria-ios tax-return work; the desktop turn served the canned non-answer) — the first non-cron sessions after weeks of cron-only windows.
  • Nothing removed or archived — no page became obsolete, and no one-way sync was run.
  • No secrets written. The gateway token pasted in the 2026-10-20 desktop session was not transcribed into any page; both the incident and current-state entries describe the request without the credential. Host names, IPs, job/incident IDs, ports and probe status codes only.

[2026-10-21] stocktake | Scheduled wiki maintenance (cron 1a8e90ea2271, 03:00 AWST)

  • Maintenance script ran clean — wiki-maintain.sh completed all three steps: skills-index regenerated (346 → 347 skills; new australian-tax-return-review under Finance), freshness audit written (raw/stocktake/freshness-20261021.txt), Quartz rebuilt (596 files) and deployed to public/, hermes-wiki-serve.service restarted.
  • infrastructure/tailscale — node table rebuilt from live tailscale status --json. The 2026-10-19 table had drifted again: octop (100.92.163.122) and springroll (100.100.153.125) were missing entirely, openmausbot / openmausbot-vm were described in prose but absent from the table body, omarchymac had gone offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on DNS name rather than the JSON HostName — the latter is ct260 for the freshrss node, and both OmniRoute nodes share the hostname omniroute, which silently collides. Now 51 non-funnel nodes (50 IPv4 peers + hermes). Container states cross-checked against live pct list on pve (100.96.100.82) and m93p (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
  • reports/uber-eats — two missing monthly reports recovered. The generator (~/.hermes/cron/scripts/generate-uber-eats-report.py) wrote only to ~/wiki/reports/uber-eats/ (the ~/wiki repo), never to the canonical agent-estate-wiki tree, so the 2026-08 and 2026-09 reports were missing from the published wiki. Root cause fixed: the script now writes to ~/workspace/agent-estate-wiki/reports/uber-eats/ and mirrors to the legacy tree. It also now emits YAML frontmatter — without it the pages landed in the "no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing months copied in and verified against ~/.hermes/uber_eats_report.json (2026-09: $51.29 / 2 orders; 2026-08: $211.16 / 7 orders). The 2026-08 previous-month column was corrected — the original run had no comparison payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change.
  • incidents/2026-08-15-skills-index-drift and incidents/2026-09-02-sftpgo-outage re-verified against live state and annotated. The skills-drift page named a regenerate-skills-index.sh that no longer exists on disk (regeneration now lives inside wiki-maintain.sh step 1/3); the SFTPGo page confirmed CT299 running and the health endpoint returning HTTP 401 (auth required, service up) — no recurrence.
  • Still stale, left alone deliberately: reports/communities-wa-social-media-audit-2026 (54d) — its source document /home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md no longer exists on disk, so the audit baseline cannot be re-verified against a live source. Flagged, not edited.
  • Fleet unchanged — 28 jobs / 25 enabled. This job (1a8e90ea2271) is active, last_status: ok, model: free-stack / provider: custom:omni-101, next run 2026-10-22 03:00.
  • No secrets written. Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.

[2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the omni-101 pin

  • New page: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday, custom:omni-101 / smart-route) failed 2026-10-21 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — a string this fleet has never produced before, and not the 503 resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session cron_dca8482e4f76_20261021_070022 wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls); incident dca848_014affeb6954 is alerted and unclosed; it was the day's only failure (171 completed / 1 failed).
  • Infrastructure: Hermes Agent updated to 0.22.3 — hermes update ran 22:47:25 → 22:57:38 AWST (action 9eed3a99e799347c691ea52a72e122a1), pre 0.22.2 / d923a5eb… → commit 5b147ab036… (detached), stamp built 2026-10-21T14:57:38Z. Stage snapshot = skipped (disabled by updates.pre_update_backup (mode: off)), so the update had no snapshot and the off-box copy path is still dead. Gateway restarted via systemd (pid 868087, phase: running, heartbeat 00:13 AWST), dashboard respawned by argv (:9119, pid 869852). Advisories printed, neither actioned: rebuild the search index (hermes sessions optimize-storage) and Preparing pinned cua-driver… (no completion line in the log). The intermediate d923a5eb rebuild was never recorded on the page, so the previously-noted v0.21.5+6169.gab86a20 (2026-10-04) was already stale — corrected.
  • Pages updated: systems/hermes-agent.md (version/upstream bullets rewritten, frontmatter dated), systems/scheduled-tasks.md (new current-count bullet; 10-21 bullet marked superseded), systems/omniroute.md (new 2026-10-21 section: three signatures, one pin), incidents/2026-10-16-omniroute-503-resource-pressure.md (window-close update; stays monitoring), incidents/2026-08-08-gitea-config-backup-push-dead.md (2026-10-21 run: backlog 75 → 76, message_id 8449), infrastructure/backups.md (76 commits; pre_update_backup: false now has a concrete exercised example), current-state.md (new dated section), index.md (new incident entry, scheduled-tasks summary, date).
  • Ledger facts as recorded this window: fleet 28 jobs / 25 enabled / 3 disabled; non-clean last_status 9 (7 error / 2 delivery_failed) with one swap — 08eb86f02d37 out (its 08:30 slot ran clean and 08eb86_472c7b609c80 resolved 08:37:21), dca8482e4f76 in; 1 incident opened / 1 closed, unclosed stays 9 (8 alerted / 1 detected); 12 sessions, all cron, no interactive sessions; 2026-10-20 settled at 169 completed / 1 unknown (the orphan) — the 10-21 audit's "168 / 0" was taken before the 23:59 slot landed, and that correction is recorded in both systems/scheduled-tasks.md and current-state.md.
  • No secrets written. Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.

[2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)

  • Ran ~/.hermes/scripts/wiki-maintain.sh after the audit so the new incident page is actually published, rather than waiting for the 03:00 stocktake.
  • systems/skills-index.md regenerated from the live tree: 348 skills (was 347) across 74 top-level directories; section order now follows the script's category-priority list. Auto-generated — no hand edits.
  • Freshness report raw/stocktake/freshness-20261022.txt: 1 stale / 95 fresh / 6 no-date of 102 pages. The only stale page is reports/communities-wa-social-media-audit-2026.md (55 d).
  • Quartz build + deploy: 273 HTML files into public/; hermes-wiki-serve.service restarted 00:18:05 AWST (the build wipes and recreates public/, so the running server always needs the restart).
  • Verified live: http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip → 200, and all four of its wikilinks resolve (systems/omniroute, systems/scheduled-tasks, systems/hermes-agent, incidents/2026-10-16-omniroute-503-resource-pressure); the new entry appears twice on the served index.

[2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)

  • systems/scheduled-tasks.md — new ## Count bullet for the 2026-10-22 window: 172 completed / 0 failed (+ this audit running), zero failures for the second time in three days; fleet unchanged 28 jobs — 25 enabled / 3 disabled; five incidents closed in-window and none opened, so unclosed rows fall 9 → 4; non-clean last_status falls 9 → 5; 27 sessions (15 cron, 3 subagent, 6 desktop, 2 acp, 1 oneshot). Previous bullet marked superseded; updated/verified_on bumped to 2026-10-23, review_after to 2026-10-30.
  • Correction recorded: a DB recount (by both started_at and claimed_at, AWST) puts 2026-10-21 at 168 completed / 1 failed, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it (floor 2026-10-08); the delta is written into the page rather than silently changed.
  • incidents/2026-08-15-skills-index-drift.md reopened (status: resolved → active, reopened_on: 2026-10-23). Evidence: the 2026-10-22 stocktake (1a8e90ea2271) ran 03:00:28 → 04:38:19 (97 m 58 s, vs 16 m 01 s on 10-21), finished completed with delivery suppressed, and committed nothing — its regenerated systems/skills-index.md (mtime 03:00, ## .Archive (41) section, 347 total) was still uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo's wiki-maintain.sh (find-based, 347 with .Archive) and the stale ~/.hermes/scripts/wiki-maintain.sh (rglob-based, 348 without). Live counts verified: filesystem 348, with symlinks 376, minus .archive/ (41) and apple/ (4) = 331, loader catalogue 329 enabled (34 hub-installed / 52 builtin / 243 local).
  • Root-cause fix applied: the repo's wiki-maintain.sh now prunes */.archive/* from the skill scan (the loader does not expose archived skills), prints the archived count as an explicit exclusion, and names hermes skills list as the authoritative loadable figure. systems/skills-index.md regenerated: 306 skills across 69 categories (was 347 counting the parked archive). Script fix and regenerated page committed together.
  • Quartz rebuild + deploy: 102 files parsed, 602 files emitted to public/, deployed, and hermes-wiki-serve.service restarted 00:14:22 AWST.
  • Freshness report raw/stocktake/freshness-20261023.txt: 2 stale / 94 fresh / 6 no-date of 102 pages — the long-standing reports/communities-wa-social-media-audit-2026.md (56 d) and, newly over the threshold, systems/model-providers.md (22 d, updated 2026-10-01). The latter needs a review pass against the live OmniRoute catalogue before its next verified_on.
  • No secrets written. Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.