Files
agent-estate-wiki/log.md
T

103 KiB
Raw Blame History

Wiki Log

Chronological record of wiki actions. Append-only.

[2026-07-22] init | Fresh Hermes Systems Wiki

  • Created technical wiki at /home/hermes/wiki
  • Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
  • Wrote SCHEMA.md with 11 operating rules
  • Wrote index.md as curated Start here dashboard
  • Reseeded system/infrastructure/runbook stubs for existing homelab
  • Git initialized, local-only
  • No secrets, passwords, tokens, or private keys written to vault

[2026-07-22] audit | External audit pass (Claude, at Anthony's request)

  • Merged duplicate pages: systems/omniRoute.md (empty stub) -> kept systems/omniroute.md (populated); repointed 5 links
  • Merged duplicate pages: runbooks/safely-update-hermes.md -> kept as runbooks/update-hermes-safely.md (fuller content); repointed 2 links
  • Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
  • Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
  • Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
  • Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
  • Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts

[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)

  • Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
  • Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
  • Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
  • Fixed a broken review_after: 0001-01-01 in decisions/keep-openrouter-as-fallback.md
  • New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
  • queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate

[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)

  • Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
  • Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
  • Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
  • Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
  • Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
  • Confirmed: Mnemosyne is, and has only ever been, this box's memory backend

[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)

  • Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
  • These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki

[2026-07-25] operations | Tool Search and gateway resilience

  • Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
  • Documented how new skill categories are derived from the first folder below ~/.hermes/skills/
  • Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
  • Recorded the I/O-controller delegation limitation so an ineffective IOWeight is not mistaken for active protection
  • Added health checks, exact effective settings, backup location, and rollback procedure
  • No secrets, tokens, or private credentials were written to the wiki

[2026-07-28] dashboard | Jobs Dashboard feature additions

  • Fixed bulk status update (payload shape mismatch: {ids, status} → {ids, updates: {status}})
  • Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
  • Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
  • Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
  • Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion Notes / JD Summary
  • Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
  • Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
  • Updated Synergy and Dept of Communities to Phone Screen status
  • Updated wiki runbook with full dashboard feature documentation and API endpoint table

[2026-07-27] incident | Job Radar search and dashboard recovery

  • Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
  • Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require MISSING_COVER_LETTER_COUNT=0
  • Recorded the production cron model (gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5)
  • Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
  • Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both :443 Serve and :8443/webhook Funnel
  • Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
  • Verified no secrets, API keys or private credentials were written to the wiki

[2026-07-29] update | Steel Browser deployment — browser backend switch

  • New page: systems/browser-backend.md — rewritten to reflect Steel Browser as primary, Camoufox as fallback
  • New page: comparisons/steel-vs-camoufox.md — comparison of the two backends
  • Updated: infrastructure/hosts.md — added CT201 and CT460 to verified hosts table
  • Updated: infrastructure/docker-services.md — added Steel Browser compose file
  • Updated: infrastructure/tailscale.md — added CT201 Steel serve routes
  • Updated: index.md — browser-backend link description
  • Updated: current-state.md — added 2026-07-29 changes section
  • Config change: browser.cdp_url set to http://100.96.244.39:9223 in ops profile

[2026-07-30] update | Open WebUI updated on CT245

  • Updated image to ghcr.io/open-webui/open-webui:main (new digest e97bf9531916, Jul 27)
  • Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
  • Restarted container after OOM hang
  • Updated infrastructure/proxmox.md

[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed

  • Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
  • Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
  • Ran mnemosyne reindex --yes to rebuild all embeddings — working memory now at 100%
  • Updated health check script (mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py)
  • Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
  • Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
  • DB grew from 8.7M to 21M with the new embeddings

[2026-08-01] incident | Mnemosyne capture pipeline repaired

  • Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no working_memory row existed.
  • Corrected stringified structured YAML (sync_roles, skip_contexts, reflect, sleep_threshold, ignore_patterns) and enabled profile isolation; hermes config check passed.
  • Verified the repair with a second CLI marker that persisted as an exact normalized [USER] database row.
  • Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route :9119; added authoritative identity/preference/workflow slots.
  • Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
  • Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.

[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation

[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync

  • Ops profile (~/.hermes/profiles/ops/config.yaml): enabled mnemosyne plugin with profile_isolation: true, allow_tool_override: false, shared_surface_read: false
  • Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
  • Verified end-to-end: memory writes land correctly in both profiles independently
  • Updated systems/mnemosyne — profile isolation section now documents ops config
  • Updated current-state — added ops profile Mnemosyne entry
  • Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
  • Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision

[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation

  • Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances
  • Reworked deepseek-flash-free and good-free on CT600: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort
  • Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
  • Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
  • Updated systems/omniroute — added Recent Changes (2026-08-04) section
  • Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated updated date

[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite

  • OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. is_active=0 does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
  • Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
  • Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
  • Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
  • Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
  • Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
  • Updated index.md — last-updated date 2026-08-04 → 2026-08-07

[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review

  • Portainer password reset documented: CT245 admin user confirmed as anthony (not admin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state.
  • SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint, // root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.

[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review

  • Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker from freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named jobs (e.g. vertex/gemini-2.5-flash, deepseek-v4-flash-free) → smart-route. Daily Marketing Job Radar → gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run (ID a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, CT333 Nanobot has live jobs.json with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
  • Updated current-state — added 2026-08-08 changes section
  • Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues

[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue

  • Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
    • TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
    • RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
  • Root cause identified: Disk at 83% (11GB free on 65GB), state.db at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
  • Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
  • Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
  • OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
  • No new pages created — all changes applied to existing pages

[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals

  • OmniRoute AutoRoute variants documented: auto/best-free (Auto Cheap) and auto/coding-free are virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: auto/best-free routed to Antigravity. Updated systems/omniroute.
  • OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains memory.provider: mnemosyne. Added to comparisons/memory-backends.
  • FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default freellmapi-auto. Added to comparisons/model-routers.
  • No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.

[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration

  • OmniRoute migrated from CT600 to CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list, grep, Tailscale DNS).
  • Updated systems/omniroute — Current Configuration rewritten for omni/CT601, added Recent Changes (2026-08-13), bumped updated/verified_on to 2026-08-13.
  • Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped updated.
  • Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
  • Updated infrastructure/hosts — OmniRoute proxy entry → omni; CT600 reference marked verified/retired.
  • Updated index — last-updated date 2026-08-10 → 2026-08-13.
  • No new pages created; all changes applied to existing pages. No secrets written.

[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated

  • Skills index drift incident: systems/skills-index.md claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating regenerate-skills-index.sh script, and adding cron job 1a8e90ea2271 (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift.
  • Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped updated/verified_on to 2026-08-15.
  • Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
  • Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
  • Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
  • Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
  • No secrets written.

[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash

  • New page: systems/nextcloud.md — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
  • Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
  • Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
  • Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
  • No secrets written.

[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk

  • OmniRoute upstream capacity pressure documented: ServiceUnavailableError: Structurally heavy chat request capacity is busy reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
  • Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
  • Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config http://omni:20128/v1 is correct — /v1 is required.
  • Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped updated/verified_on to 2026-08-16.
  • Updated current-state — added Recent Changes (2026-08-16) section, bumped updated to 2026-08-16.
  • Updated index.md — bumped last-updated date to 2026-08-16.
  • No new pages created. No secrets written.

[2026-08-18] incident | Mnemosyne capture pipeline broken (active)

  • New page: incidents/2026-08-18-mnemosyne-capture-pipeline — working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
  • Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped updated/verified_on to 2026-08-18, added cross-reference to Related.
  • Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped updated to 2026-08-18.
  • Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
  • Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from active to resolved (that earlier incident was fixed).
  • No secrets written.

[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history

  • Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
    • wiki-stocktake-maintenance (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from cx/gpt-5.6-luna → go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.
    • CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
    • Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
  • Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped updated/verified_on to 2026-08-23.
  • Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped updated/verified_on to 2026-08-23.
  • Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
  • Updated index.md — bumped last-updated date to 2026-08-23.
  • No secrets written.

[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status

  • New page: systems/ignis.md — documented Ignis browser-native Obsidian (Docker ignis-subpath:0.8.9, port 8080, Tailscale Serve at hermes.kangaroo-eel.ts.net/ignis). Volume mount: /home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active.
  • Updated systems/model-providers — added command-code provider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumped updated/verified_on to 2026-08-19.
  • Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider, deepseek-flash-mix combo, and Tavili search proxy status. Bumped updated/verified_on to 2026-08-19.
  • Updated infrastructure/docker-services — added Ignis compose file and service definition.
  • Updated infrastructure/hosts — added Ignis to CT460 row.
  • Updated systems/obsidian — added Ignis as browser interface, corrected vault path to agent-estate-wiki/, added Quartz static site reference.
  • Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
  • Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
  • No secrets written.

[2026-08-26] update | Nightly wiki audit — Cron job reconciliation

  • Updated systems/scheduled-tasks — reconciled live hermes cron list (27 jobs) against wiki table (29 entries). Removed: Workday start reminder (no longer in live list), CT 221 Miner Watchdog (job d5516d0a5147 removed; CT221 status unknown), Review Agent Estate Phase One (one-time job completed 2026-08-08). Added: weekly-save-restricted-bot-restart (0 3 * * 0, local, no-agent). Updated count from 29 to 27.
  • Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
  • Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
  • Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
  • No secrets written.

[2026-09-02] incident | SFTPGo outage on CT299

  • New page: incidents/2026-09-02-sftpgo-outage.md — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), tailscaled died; PVE pct start failed due to 5 stale lxc-attach processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop 299 --skiplock + pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
  • Updated current-state — added Recent Changes (2026-09-02) section, bumped updated/verified_on to 2026-09-04.
  • Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
  • No secrets written.

[2026-08-28] update | Department of Communities role and social-media audit

  • New page: reports/communities-wa-social-media-audit-2026 — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
  • Updated systems/current-profile — recorded current employment context and linked the report.
  • Updated index.md — added the report to the Reports section and refreshed the last-updated date.
  • No secrets written.

[2026-09-20] update | Nightly wiki audit — No changes in past 24h

  • No new pages created — no new systems, decisions, or incidents in the past 24 hours.
  • No existing pages updated — all wiki content remains current.
  • Freshness audit run: freshness-20260920.txt generated at 03:01 AWST (scheduled cron).
  • Skills index auto-regenerated: systems/skills-index.md updated at 03:01 AWST (scheduled cron 1a8e90ea2271).
  • Scheduled tasks reconciled: systems/scheduled-tasks.md updated at 03:02 AWST — 26 active jobs verified from live hermes cron list.
  • No secrets written.

[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation

  • New page incidents/2026-09-28-hermes-managed-runtime-deps — the 2026-09-28 → 30 cron-fleet outage: jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused by ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in the uv-managed runtime venvs under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (all 10 venvs, pip install --target), run every 15 min by job 96eb95ce12f2.
  • Updated systems/scheduled-tasks.md — table rebuilt from live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to d40085da631c, f77b7e46bd09, 0a7f933d06e2. Known Issues refreshed: job-radar 403 MODEL_NOT_IN_PLAN (streak 5), Telegram thread-delivery failures, weekly-save-restricted-bot-restart script restored, residual pre-fix dispatch errors on cbfe434a3b5e/1a8e90ea2271.
  • Updated systems/hermes-agent.md — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model custom → http://100.78.63.20:20128 / smart-route; skills 330 → 341; cron count 29 → 30.
  • Updated current-state.md — _config_version: 46 and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
  • Updated runbooks/job-radar.md — corrected primary model row (custom:omni-101 / gpt-5.6-terra via command-code) and added the 403 MODEL_NOT_IN_PLAN failure mode with recovery steps.
  • Updated systems/messaging-integrations.md — Telegram allowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
  • Updated index.md — last-updated to 2026-09-30, cron count to 30, new incident listed.
  • No secrets written.

[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene

  • No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
  • Updated systems/scheduled-tasks.md — table and counts re-verified against live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — dca8482e4f76 (403 MODEL_NOT_IN_PLAN, ×5), 1a8e90ea2271 (HTTP 410, model retired upstream, ×7), fad5a86979d4 (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: 1b23c042e17d and c8d52904e1d6 delivery targets now telegram:-1004321904721:102; c0a9fa314542 flagged for a revoked Telegram bot token; bare-telegram targets noted on 0e753e9720fd, fad5a86979d4, 066bb5191280. Known Issues restructured into Resolved / Open / Historical.
  • Updated systems/messaging-integrations.md — new Delivery target semantics section: a bare deliver: "telegram" resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails with Message thread not found and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
  • Updated systems/model-providers.md — provider stack rebuilt from live config.yaml: 5 custom providers (omni-101 5,384 models, Omniroute:20128 5,398, litellm 1,728, command-code 71, guacano 46); custom:omni recorded as removed. Default model corrected resilient-free → smart-route. New Retired / Unavailable Models table (retired qwen3.5:397b, plan-gated gpt-5.6-terra, catalog-missing deepseek-v4-flash) with the lesson that a raw model pin never self-recovers.
  • Updated systems/mnemosyne.md — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as [USER] [Anthony Martin] memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
  • Updated systems/hermes-agent.md — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
  • Updated index.md and current-state.md — last-updated bumped to 2026-10-01 with a Recent Changes block.
  • Vault sync: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under wiki/, not at the tree root, and index.md sits at the vault root.
  • No secrets written.

[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split

  • New page: incidents/2026-10-01-cliproxy-connection-auth-expired.md — job 06a0a1c8704f (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard (provider custom:Omniroute:20128, model cliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d, 79f9bb41ecf5).
  • Updated systems/scheduled-tasks.md — counts re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01: 06a0a1c8704f (HTTP 401), 0e753e9720fd (HTTP 400 — same NVIDIA model pin as the already-failing fad5a86979d4, confirming the shared-pin diagnosis), 066bb5191280 (delivery failed; report itself generated). Streaks advanced: dca8482e4f76 ×5→×6, 1a8e90ea2271 ×7→×8, fad5a86979d4 ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
  • Updated systems/messaging-integrations.md — corrected a wrong page fact. The page previously stated that a bad thread_id always loses the message. The 2026-10-01 records show two variants: Message thread not found (hard failure, message lost — the 2026-09-30 case) and configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id (a thread-less fallback send does occur, yet the run is still recorded delivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegram jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
  • Updated systems/current-profile.md — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no ~/.hermes/profiles/default/ directory (the default profile is the top-level config; ops is the only named sub-profile), the model default is smart-route (not freellmapi-auto), the default provider is custom → http://100.78.63.20:20128 / OmniRoute (litellm is now only the last fallback entry), and the config is _config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
  • Updated current-state.md — new Recent Changes (2026-10-02) block; frontmatter dates bumped.
  • Updated index.md — last-updated to 2026-10-02; cron line now notes the failing count.
  • Structural finding — the wiki exists as two trees and they have drifted both ways. /home/hermes/workspace/agent-estate-wiki is canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads (pages under wiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions — systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md and reports/communities-wa-social-media-audit-2026.md exist only in canonical — so canonical index.md currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), failing set still 6 but the composition moved. 1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard that the cliprox incident predicted — making it 2 of the 3 cliprox/terra jobs confirmed broken, which retires any "one-job fluke" reading. 79f9bb41ecf5 is the last untested job on that pin (next due 2026-10-04 18:00). Streak recount against executions.db: dca8482e4f76 ×6→×7, but 1a8e90ea2271 ×8→×7 — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). Resolved: 1b23c042e17d's stale delivery_failed flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). Downgraded: the GitHub Trending Radar revoked-token claim is now "unverified at last run" — c0a9fa314542's 2026-09-30 19:22 run completed and delivered. New Known Issue recording that a delivered notice is not a successful job (both cliprox failures read delivery_outcome: delivered on a failed status).
  • New page incidents/2026-08-26-work-outlook-basic-auth-disabled.md — the work Outlook mailbox has been unreachable since 2026-08-26: himalaya is configured for password auth while the tenant advertises only XOAuth2, so himalaya -a work dies with login not supported. TCP/TLS to outlook.office365.com:993 is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
  • Updated systems/messaging-integrations.md — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, work unreachable; frontmatter updated/verified_on bumped. Added the tooling consequence: email/mail-lookup resolves the account before searching, so work-mail requests hit this wall and must be reported as account unreachable, not "no results found".
  • Updated systems/hermes-agent.md — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to six jobs across three model faults plus one credential fault, and a second entry added for the stock-take job being down; the Email: connected bullet corrected.
  • Updated current-state.md and index.md — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
  • systems/skills-index.md regenerated 341 → 344 — including a new skill, fleet-wide-failure-triage (devops). This was done by hand: job 1a8e90ea2271 is still blocked on HTTP 410 (retired qwen3.5:397b) and cannot self-recover. wiki-maintain.sh itself is healthy and was run manually, also writing raw/stocktake/freshness-20261003.txt — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under wiki/; index.md sits at the vault root.
  • No secrets written; all credential-shaped values in config remain redacted.

[2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair

Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.

New pages

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md — mnemosyne_recall had been scoring keyword-only: fastembed was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 without raising an error. Repaired 2026-10-03; the runtime guard now asserts fastembed + FASTEMBED_CACHE_PATH + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
  • incidents/2026-10-03-ct450-hermes-update.md — CT450 (hermesagent) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (ssh root@100.96.100.82 → pct exec 450 -- …) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).

Updated

  • current-state.md — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the mnemosyne doctor / sqlite_vec misread gotcha.
  • systems/mnemosyne.md — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job b63ed5817212 documented.
  • systems/hermes-agent.md — version corrected to v0.21.5+6169.gab86a20 (was v0.21.5+3837.gbfda74c), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no profiles/default/; ops is the only named sub-profile), Known Issues bullet added.
  • systems/memory-backend.md — Known Issues bullet for the FTS-only degradation.
  • systems/scheduled-tasks.md — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), 79f9bb41ecf5 due 2026-10-04 18:00; bare-telegram delivery warnings confirmed live.
  • agents/hermes-production.md — version, Python, profile, provider routing, cron count and messaging refreshed; stale custom:litellm / freellmapi-auto references replaced.
  • agents/hermes-chaos.md — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
  • infrastructure/hosts.md — CT450 row updated to v0.21.5.
  • runbooks/update-hermes-safely.md — new "Second instance — CT450" section (access path + pre-flight checks).
  • runbooks/provider-health.md — stale v0.19.0 version reference corrected.
  • index.md — last-updated bumped; both new incidents linked; cron-fleet line re-dated.

Note — mnemosyne doctor on this host prints vec0: present_but_unloadable and marks the vector tiers unavailable. That is a Python-3.11-legacy-venv artefact, not the gateway's state. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py.

[2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7

Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.

  • Updated systems/scheduled-tasks.md — fleet re-verified against live ~/.hermes/cron/jobs.json + hermes cron list: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 6 → 7. The one new failure event in the window is 79f9bb41ecf5 (Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identical HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. It was the last untested job on the cliprox/terra pin, so that pin is now 3 of 3 confirmed broken — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271) and the Known Issues Open list all updated; frontmatter dates bumped.
  • Updated incidents/2026-10-01-cliproxy-connection-auth-expired.md — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still Open — needs the OmniRoute connection re-authenticated before any repin.
  • Updated current-state.md — new Recent Changes (2026-10-05) block: the cliprox 3/3 settlement, 1a8e90ea2271's repeat 410 on 2026-10-04, confirmation that no interactive sessions occurred in the window (all 10 active sessions were cron), and a streak-counting caveat.
  • Updated index.md — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
  • Streak-number caveat recorded: hermes cron list's "N failures in a row" is the failure_streak counter in jobs.json, not a DB count. ~/.hermes/cron/executions.db now retains only rows from 2026-09-30 onward (1,001 rows total), so executions.db recounts undercount older streaks — the earlier DB-based "×7" for 1a8e90ea2271 undercounts against the live ×11. Quote the live counter and note the retention floor.
  • Vault mirror: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source — wiki/systems/scheduled-tasks.md, wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md, current-state.md, index.md, log.md.
  • No secrets written; all credential-shaped values remain referenced by connection ID only.

[2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0

Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.

New

  • incidents/2026-10-05-dead-model-pin-repin.md — the seven-job dead-model-pin repin. Records the four dead pins (cliprox/terra HTTP 401 ×3, ollama-cloud/qwen3.5:397b HTTP 410 ×1, nvidia/deepseek-ai/deepseek-v4-flash HTTP 400 ×2, command-code/gpt-5.6-terra HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (free-stack ×6, smart-route ×1 on custom:omni-101) over another raw model name.

Updated

  • systems/scheduled-tasks.md — Count block rewritten: 0 failing (was 7), with all 30 failure_streak counters at 0; four jobs carry a stale pre-repin last_status: error that clears on their next scheduled run. Seven job-table rows updated (06a0a1c8704f, 1b23c042e17d, 79f9bb41ecf5, 1a8e90ea2271, fad5a86979d4, 0e753e9720fd, dca8482e4f76), plus c0a9fa314542. New ### Resolved 2026-10-05 subsection in Known Issues; five resolved bullets removed from ### Open, leaving bare-telegram targets, the two-wiki-tree drift, the work-mailbox block, the stale custom:omni key and the delivered-notice rule as the standing open set.
  • index.md — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.
  • current-state.md — new Recent Changes (2026-10-06) block; frontmatter dates bumped.

Findings worth the ink

  • dca8482e4f76 (Daily Marketing Job Radar) delivered to topic 1030 at 23:56 — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for 79f9bb41ecf5 (23:39) and 06a0a1c8704f (23:45).
  • GitHub Trending Radar's revoked-token diagnosis is dead. c0a9fa314542 ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No getMe → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright.
  • delivery_outcome: delivered still does not mean the job worked. Every pre-repin failure recorded delivered, because the failure notice is itself the deliverable. Judge on the execution status column.
  • A stale notice opened the review. Anthony was prompted by a 85edf6772bca notice that had sat in topic 346 since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.
  • Window tally (executions.db, 2026-10-05 00:00 → 2026-10-06 00:15): 185 completed, 3 failed (all pre-repin, 03:00–07:00), 2 unknown (aborted first attempts of 79f9bb41ecf5 and dca8482e4f76 at 23:33/23:48, superseded by their successful re-runs) and 1 running (this audit).

Vault mirror: the new incident page plus systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day

Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — Count block corrected: the stale pre-repin last_status: error set is now three (1b23c042e17d, fad5a86979d4, 0e753e9720fd), not four — 1a8e90ea2271 cleared its own when it ran at 03:00. Added a Re-verified 2026-10-07 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 0 failed / 1 running; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
  • index.md — last-updated → 2026-10-07; cron-fleet line re-dated (0 failing as of 2026-10-07, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
  • current-state.md — new Recent Changes (2026-10-07) block; frontmatter dates bumped.

Findings worth the ink

  • 1a8e90ea2271 ran itself at 03:00 and proved the repin. wiki-stocktake-maintenance completed 2026-10-06 03:00:31 → 03:03:32 on custom:omni-101 / free-stack — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired qwen3.5:397b). It regenerated systems/skills-index.md (344 → 346 skills), wrote raw/stocktake/freshness-20261006.txt and committed to Gitea as 33f4e95. Its four HTTP-410 incidents (6df957a05ec6, d284168d3663, cda67de5b3e3, e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. The daily manual wiki-maintain.sh run can stop.
  • Vault mirror drift found and partly repaired. systems/skills-index.md in /home/hermes/wiki/agent-estate-wiki/ was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
  • Window tally (executions.db, 2026-10-06 00:00 → 2026-10-07 00:00): 176 completed, 0 failed, 1 running (this audit) — 177 rows. 14 sessions, all cron; no interactive (non-cron) sessions, and no new incident opened.

Vault mirror: systems/scheduled-tasks.md, systems/skills-index.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by connection ID only.

[2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings

Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.

Updated

  • systems/scheduled-tasks.md — new Re-verified 2026-10-08 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 completed / 1 failed / 1 running; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New Resolved 2026-10-07 Known Issues section, plus two new Open items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.
  • index.md — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.
  • current-state.md — new Recent Changes (2026-10-08) block; frontmatter dates bumped.

Findings worth the ink

  • The incident machinery had its first real workout since 2026-09-30 — and it worked. 62619aa6bcb3 (DHL Tracking) failed its 08:00 run inside ensure_browser() with TimeoutError: timed out on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident 62619a_f485ccfd4a0d (failure_type: timeout) was opened 08:00:52, alerted 08:00:58 and auto-resolved 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed.
  • The tracked parcel has arrived. The 14:00 run reported waybill 8052838382 as Delivered, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed.
  • Stocktake ran automatically for the second consecutive day (1a8e90ea2271, 2026-10-07 03:00, commit 8c63ef2; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once.
  • Two real defects surfaced from the stocktake's own log. (1) Step 3 of wiki-maintain.sh — the Quartz rebuild — is skipped on every run: there is no quartz.config.ts and no npx, so public/ is never refreshed and the published site silently ages. (2) skills-index under-reports by one: 347 SKILL.md files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md (the other duplicated basename, hermes-api-vault, survives because its two copies sit in different categories). Both are now Open items in systems/scheduled-tasks.
  • Unchanged, still carrying a stale last_status: error: 1b23c042e17d, fad5a86979d4, 0e753e9720fd — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-telegram targets and the custom:omni stale-key references are also unchanged.

Vault mirror: systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

[2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open

Window audited: 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days.

Pages written

  • incidents/2026-10-09-cron-provider-timeout-storm.md — new.
  • incidents/2026-10-09-memory-recall-relevance.md — new.
  • systems/scheduled-tasks.md — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
  • systems/mnemosyne.md — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
  • current-state.md — Recent Changes (2026-10-09) block; frontmatter dates bumped.
  • index.md — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
  • systems/skills-index.md + raw/stocktake/freshness-20261008.txt — the stranded 2026-10-08 03:01 stocktake artifacts committed.

Findings worth the ink

  • The fleet had its worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight remain open. The watchdog (96eb95ce12f2) failed twice as a cascade — it exits 1 when it finds ERROR lines in agent.log, i.e. it reported the storm rather than suffering it.
  • The 2026-10-08 window was never audited — this job's own 23:59 run failed with the same timeout (incident cbfe43_fa81052baaf7) and committed nothing.
  • The stocktake has a two-day gap. 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no freshness-20261009.txt exists. 1a8e90ea2271 is the one job still at failure_streak: 2.
  • An instruction was reported done and was not. Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that 62619aa6bcb3 was "stopped and deleted". It is still enabled (paused_at: null) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
  • Memory recall relevance — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is not in live config (memory.mnemosyne carries no recall-weight keys; no top-level mnemosyne: key).

Vault mirror: the touched pages mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.

No secrets written; credential-shaped values referenced by job ID and connection ID only.

2026-10-10 — Stocktake (manual run, wiki-maintain.sh)

  • systems/skills-index.md regenerated — 346 skills / 70 categories (unchanged count; frontmatter updated/verified_on refreshed to 2026-10-10). Live on-disk count is 347 SKILL.md; the one-skill gap is the known de-dup of productivity/obsidian-git-sync/obsidian-git-sync/.
  • Freshness audit → raw/stocktake/freshness-20261010.txt: 62 stale (≥21d), 23 fresh, 7 no-date of 92 pages. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No freshness-20261009.txt exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands.
  • current-state.md — new 2026-10-10 section. The 2026-10-09 timeout storm window re-tallied from executions.db (2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163 completed, 15 failed — the prior audit recorded 162/15+1 running; the stray running row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).
  • DHL monitor re-probed, still enabled. 62619aa6bcb3 — enabled: True, paused_at: null, 0 8,11,14,17,20 * * *, last_status: ok. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. No cron mutation performed by this audit — retirement stays open item #1.
  • infrastructure/tailscale.md verified against tailscale serve status — all 13 tailnet Serve routes plus the :8443/webhook Funnel match the page. No drift; not edited.
  • Not verified / left alone: the Quartz rebuild step (no quartz.config.ts, npx cannot fetch quartz — public/ was NOT refreshed, confirming the carried open item); docker ps (permission denied on /var/run/docker.sock from this session, so infrastructure/docker-services.md could not be probed and is left stale); the 80d 2026-07-22 cluster (no live source to check against).
  • Sessions in the 2026-10-09 window (state.db, epoch REAL): 19 total — 6 cron, 11 ios, 1 talaria-ios, 1 telegram. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the delivered count includes failure notices).

No secrets written; job/incident IDs only.

[2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-11T00:19). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 174 completed / 0 failed. Non-clean last_status 10 -> 5. Incidents: 7 closed during 10-10, 5 open (3 jobs). Open item #1 closed: the DHL monitor 62619aa6bcb3 is retired and verified absent from jobs.json by ID lookup (the 10-07 and 10-09 "deleted" claims were false).

Runtime/memory. The 2026-09-28 managed-runtime-dependency class recurred in the gateway env: mnemosyne (+native deps) and elevenlabs were missing from the committed uv env 795020dbf24343cd9c52df24723dca83 that activate_dependencies() loads. Repaired 21:26-21:34 and verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.

New page. systems/paseo - Paseo daemon v0.11.2 on 0.0.0.0:6767 (user unit paseo-daemon.service); recorded the daemon run vs daemon start supervisor trap, the /usr/bin/paseo 0.7.2 vs daemon 0.11.2 version split, and the truncated-cline false provider fault.

Platform. Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 answers neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.

Pages touched: systems/paseo.md (new), systems/scheduled-tasks.md, systems/hermes-agent.md, systems/mnemosyne.md, systems/messaging-integrations.md, incidents/2026-09-28-hermes-managed-runtime-deps.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-11] stocktake | Daily maintenance + live drift corrections

  • Ran wiki-maintain.sh: skills-index regenerated (346 skills / 70 categories), freshness audit written to raw/stocktake/freshness-20261011.txt (62 stale / 24 fresh / 7 no-date of 93 pages)
  • OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed. Verified via pct list, pct config, ss and tailscale status: CT600 omniroute (100.88.81.19) is RUNNING and serving :20128; CT601 omni (100.93.204.4) is STOPPED with onboot=0. Hermes resolves custom:omni → omniroute:20128 → CT600, not CT601. systems/omniroute.md rewritten with the live node table.
  • Third OmniRoute node found: omniroute-1 at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider omni-101 (5384 models, default smart-route, carries skills_hub). Not in pct list — identity unconfirmed.
  • Ignis duplicate found: a second ignis-ignis-1 container runs on CT200 (image ignis-server:latest, host port 8092, vault /opt/ignis/vaults), separate from CT460's ignis (port 8080, vault /home/hermes/wiki/agent-estate-wiki). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them.
  • Tailscale route table corrected against live tailscale serve status: /agent-estate target is bare http://127.0.0.1:8769 (not .../8769/agent-estate-role-dashboard.html); previously-missing tailnet-only :9443 → 127.0.0.1:5173 documented. CT201 routes refreshed (8085 targets + /steel 5173 + /steel-api 3000).
  • Proxmox inventory rebuilt from live pct list + qm list: added CT475 agentzero and VM110 omarchy; corrected 282 ubuntu-server-xfce to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). VM403 open-claw and VM401 hermes-old are absent from qm list — flagged unconfirmed rather than guessed.
  • Nanobot verified on CT333: gateway nanobot-gateway (PID 2259048) listening on 8765/8766; NOT a systemd nanobot.service/gateway.service (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified.
  • Docker services page corrected: the old "Docker runtime not available" note was a CT460 socket-permission artifact (/var/run/docker.sock is root:docker srw-rw----, hermes user not in group docker, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo).
  • Flagged for follow-up, root cause not investigated: mycompose-db-1 on CT200 is in Restarting (1) — crash-looping.
  • Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
  • Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.

[2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)

Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-12T00:10). Day tally: 10-07 177/1, 10-08 178/2, 10-09 163/15, 10-10 175 completed / 0 failed (finalised), 10-11 171 completed / 0 failed + 1 running — two consecutive clean days. Non-clean last_status steady at 5; 0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected).

Corrected a false alarm instead of logging a fault. The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a naive-UTC vs AWST comparison error — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in systems/mnemosyne and systems/scheduled-tasks so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.

New content. queries/retained-research-answers was a 2026-07-22 stub and is now a real page — the 2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; -CompressLevel 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage was undocumented; the live datastore table plus host mount points are now in infrastructure/proxmox (/mnt/ssd0 93 % / ~63 GiB free is the pressure point).

Stale labels fixed. incidents/2026-08-18-mnemosyne-capture-pipeline carried status: active from August despite capture being verifiably healthy — now superseded, with the live evidence and a caution about the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open, and the skills-index blurb no longer claims no regeneration since 2026-10-07.

Verified, not changed. The blocked-page-recovery lesson that session reported patching (recovered: false = archive routes failed, not task failed; API-first rung) is present in the live skill.

Pages touched: systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/proxmox.md, queries/retained-research-answers.md, incidents/2026-08-18-mnemosyne-capture-pipeline.md, current-state.md, index.md, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections

Script run. wiki-maintain.sh completed all three stages: skills-index regenerated from the live skill tree (346 skills across 70 categories, unchanged count), freshness audit written to raw/stocktake/freshness-20261012.txt (54 stale / 32 fresh / 7 no-date of 93 pages), and the Quartz step took its documented no-quartz.config.ts path (build skipped, public/ unchanged — expected, not a failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and written its artifacts, so results were read from the report rather than assumed.

Drift corrected from live probes (2 pages):

  1. systems/nanobot — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on CT333 is back up, HTTP 200 on http://100.122.189.81:8766/. Port 8900 — the endpoint the disabled OmniRoute nanobot provider connection uses — is still closed, so status stays degraded rather than flipping to active. SSH to CT333 is unreachable from CT460, so whatsapp-bridge.service was not re-verified.
  2. systems/home-assistant — carried confidence: low plus a page-wide "cannot confirm HA runs" caveat from 2026-07-22. Live: HTTP 200 on http://100.68.139.5:8123/, Tailscale node homeassistant active and offering an exit node, VMID 100 running per qm list. Confidence raised low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).

Verified and left stale. systems/openrouter and systems/nextcloud were probed shallowly and nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.

Pages touched: systems/nanobot.md, systems/home-assistant.md, systems/skills-index.md, raw/stocktake/freshness-20261012.txt, log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes

Window. 2026-10-12 00:00 → 23:59 AWST (sessions from ~/.hermes/state.db, cron from ~/.hermes/cron/executions.db + cron_incidents; live probes 2026-10-13 00:05–00:45). One former compaction of this job left part of the review written but the log entry and cross-page updates unfinished — completed here.

Headline 1 — a clean run hid a dead credential. c0a9fa314542 (GitHub Trending Radar — Weekly Roundup, Mondays 09:00) reported completed with delivery_outcome: delivered on 2026-10-12, but its fetch-then-send script died at digest.py send(). Live probe of the bot token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 from getMe — the token is a valid 46-character shape, i.e. revoked, not malformed. The fetch path is healthy (100 repos, 26 cache files in 24 h). Critically, this supersedes the 2026-10-06 entry on current-state, which concluded the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in systems/scheduled-tasks: a completed run does not clear a credential fault unless the run exercised the same call that failed, and green cron status is not delivery proof. (Credential was present; value neither recorded nor repeated.)

Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new). The 2026-10-12 07:01 Daily Marketing Job Radar run drove camofox-browser.service to 2.5G, swap fully consumed, available: 0B, 383 tasks against the drop-in ceiling MemoryHigh=2G / MemoryMax=3G / MemorySwapMax=1G. REST /tabs and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at 07:23:50, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis: workload exceeded the cgroup, not a leak — OOMPolicy=kill never fired.

Headline 3 — systems/browser-backend corrected against live probes. Prior revisions were wrong on three counts: the REST API listens on :9377 only (:9093 closed, :9378 does not exist — REST and WebSocket are the same service, not two), Camofox runs as a systemd --user unit on the Hermes host (CT460), not CT450, and the containment drop-in was undocumented. current-state.md also still described the old port model; both pages now match the probe.

Cron state. 2026-10-12: 174 completed / 0 failed / 1 running (this audit, cbfe434a3b5e, started 23:59:41) — third consecutive clean day; 2026-10-11 finalised at 172 / 0. Retention floor moved to 2026-10-07 08:15. Fleet 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3 (0e753e9720fd error, 1b23c042e17d error, 066bb5191280 delivery_failed). fad5a86979d4 (Facebook Birthday Weekly Registry Sync) and dca8482e4f76 (Daily Marketing Job Radar) both cleared on 2026-10-12 — the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.

Incidents — open 5 → 2. Closed on 2026-10-12: fad5a8_a13bbc837e1c + fad5a8_c80b59a907f0 at 03:05:39 (the two deepseek-v4-flash HTTP-400 rows, cleared by the registry sync's clean run) and dca848_23fcd2b18aaa at 07:33:41. Remaining: 1b23c0_c35703f637b8 (alerted — weekly job, cannot clear before Fri 2026-10-16 08:00) and 0e753e_e7f8194a7d06 (detected since 2026-10-01 — the job has no retained executions, so it cannot clear until 2026-11-01). → incidents/2026-10-09-cron-provider-timeout-storm updated from "8 still open" to 11 of 12 closed with the full closure timeline; the pattern recorded there is that each incident clears only on the next successful run of its own job, so weekly jobs carry their rows for up to a week after the fault is gone.

Mnemosyne (2026-10-12 12:02 health run, green). Working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 % — both above the 95 % floor, no reindex; orphan embeddings 0; FTS5 exactly aligned; 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → systems/mnemosyne.

Sessions. 18 in the window, all cron, zero interactive. The 03:00 stocktake 1a8e90ea2271 ran clean and pushed 0eee5f1 (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 categories — the known 347-vs-346 on-disk off-by-one persists).

Pages touched: current-state.md (new Recent Changes section + superseded-correction tag), systems/scheduled-tasks.md (frontmatter, job inventory, non-clean statuses, re-verified history, five job rows, ### Resolved 2026-10-12, corrected incident IDs, ### Open additions, new ## Re-verified 2026-10-13 block), systems/browser-backend.md (port + host + containment rewrite, history entries), systems/mnemosyne.md (12:02 coverage), incidents/2026-10-09-cron-provider-timeout-storm.md (status + full closure table + timeline), index.md (date, four blurbs), log.md.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential

Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.

Cron. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight → the day finalises at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 — the first failed run since 2026-10-09 (which recorded 15). Retention floor moved to 2026-10-08 07:11 (MAX_TERMINAL_EXECUTIONS = 1000; 1,001 rows = 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads 126 / 1, partial).

Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 → 03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent, alerted_at: NULL — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three on the opencode path (10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode; 10-09 timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00 run. → incidents/2026-10-05-dead-model-pin-repin

Finding 2 — the failure stranded its own generated artifacts (now committed). Step 1 (skills-index) and step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working tree: systems/skills-index.md (modified, date-only: updated/verified_on 10-12 → 10-13; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked; 48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit. Consequence: the stocktake published nothing on 10-13 — and because the agent row is what commits, even a perfect script run leaves nothing behind when the LLM turn dies.

Delivery. 159 suppressed / 9 delivered / 1 no-record. Delivered: Daily Marketing Job Radar, Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday, Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (0e753e9720fd, 066bb5191280, both Telegram thread_id 34795) are unchanged since 2026-10-01.

Mnemosyne. Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no reindex was needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than bare; the delivery layer still suppressed it (delivery_outcome: suppressed) — the matcher tolerates wrapping rather than requiring the literal token as a strict control. → systems/mnemosyne

Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive session.

Fleet. 29 jobs (26 enabled / 3 disabled), pins unchanged; non-clean last_status 3 → 4 (cleared fad5a86979d4 10-12 03:05 and dca8482e4f76 10-13 07:02; added 1a8e90ea2271); open incidents 2 → 3 (1 alerted, 2 detected, 8 closed, 66 resolved; 77 rows).

Pages touched: systems/scheduled-tasks.md (frontmatter, retention floor, non-clean list, ## Re-verified 2026-10-14), systems/skills-index.md (committed stocktake artifact), current-state.md (new Recent Changes + date), index.md (date + two blurbs), log.md.

Vault mirror. The 2026-10-13 audit was never mirrored to /home/hermes/wiki (last mirror bb09581, 2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus skills-index.md, whose vault copy was still the 2026-10-08 version. The vault's own raw/stocktake/ still holds only the 2026-10-03 report.

No secrets written; job/incident IDs, paths and tailnet addresses only.

[2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt

Ran wiki-maintain.sh (03:01:09 → 03:01:10 AWST): step 1 regenerated systems/skills-index.md (346 skills / 70 categories), step 2 wrote raw/stocktake/freshness-20261014.txt (48 stale / 38 fresh / 7 no-date of 93 pages), step 3 skipped the Quartz rebuild again (no quartz.config.ts, no npx) so public/ is untouched — the known carried item.

Finding 1 — a second, unclustered Proxmox hypervisor exists: m93p (100.104.51.98), PVE 9.2.0 (kernel 7.0.14-12-pve). pvecm nodes is empty on both pve and m93p — no cluster — and guest IDs collide between them (m93p CT221 is gitea; pve CT103 is cliproxyapi; CT100 and CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the hypervisor port :8006 returned only m93p and pve. m93p had been sitting in the Tailscale node list for weeks as an unremarkable online peer, with no wiki page recording it as a hypervisor. Live guests recorded from a single pct list / qm list pass.

Finding 2 — four guests the wiki placed on pve live on m93p. CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are absent from live pct list on pve, yet every one of the four services answers. /etc/pve/lxc/ on pve contains only 260.conf.bak and no vzdump task log references CT 221/241/245/260 — so they were not deleted, they were never on this host. The 2026-10-11 inventory was wrong and every audit since carried it forward.

Finding 3 — the tailnet inventory had drifted badly. The page listed 30 nodes; live tailscale status reports 45 non-funnel nodes (plus 23 funnel-ingress-node IPv6 peers, now noted and deliberately excluded). Fifteen nodes were missing, including m93p, pve, omniroute-1, paseo, qwenpaw, rustdesk, vaultwarden, social-dashboard, openweb, omarchybook, sftpgo, rclone, omni and runtipi. Several states were wrong too (anthonys-mac-mini is online on a different IP; omarchy-macmini / omarchy-quattro are not live peers; anthonys-iphone is online, not active). The CT460 serve routes re-verified exactly as recorded — 13 routes plus :8443/webhook Funnel and :9443 tailnet-only — so that table needed no change.

Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling: camofox-browser.service now reports Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M) with 16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the fault, as diagnosed last cycle.

Also verified, unchanged: Mattermost CT255 stopped (peer offline 24d); VM 282 ubuntu-server-xfce is now running, not stopped as recorded.

Cron (verified from executions.db / jobs.json, 03:05): 2026-10-13 = 168 completed / 1 failed (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress 23 completed / 0 failed with the 03:00 stocktake 1a8e90ea2271 still running at probe time. Open incidents 3 — 1a8e90_d11698bd7275 (detected, 10-13 opencode 404), 0e753e_e7f8194a7d06 (detected since 10-01, next slot 2026-11-01), 1b23c0_c35703f637b8 (alerted since 10-09, next slot Fri 2026-10-16 08:00). Non-clean last_status: 4 (unchanged). Fleet 29 jobs — 26 enabled / 3 disabled.

Pages touched: infrastructure/proxmox.md (two-host rewrite, m93p inventory, four-entry correction, storage, VM 282), infrastructure/tailscale.md (45-node inventory + funnel-peer exclusion note, frontmatter), infrastructure/network-map.md (m93p entry, frontmatter), current-state.md (new Recent Changes block), index.md (proxmox/tailscale blurbs + header warning), systems/skills-index.md (stocktake artifact), raw/stocktake/freshness-20261014.txt (new), log.md.

Not verified, left as stale: the remaining 46 stale pages were not probed against a live source, so they stay flagged rather than rewritten. Not re-verified this pass and still single-host in framing: systems/omniroute, infrastructure/hosts, systems/model-providers, infrastructure/docker-services — m93p also runs its own omniroute (CT101), so the OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.


[2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node

Ran 2026-10-16 00:0x–00:3x AWST. Scope is two windows, not one: the 2026-10-14 23:59 Wiki Audit (cbfe434a3b5e) failed on the fault below, so the 10-14 window was never covered.

Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).

  • Root cause. httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 — the omniroute-1 node, provider custom:omni-101 (confirmed in config.yaml: custom provider name: omni-101, base_url: http://100.78.63.20:20128). It surfaced as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. — the exact string recorded on every failing execution row. 328 Errno 111 lines in the live errors.log, 504 across the rotations.
  • Impact. 10-14 = 160 completed / 12 failed; 10-15 = 151 completed / 24 failed (+1 in flight). Against 10-12 175/0, 10-13 167/1. 36 failure rows across 13 jobs, but only 24 are genuine model-call failures on 12 jobs — the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
  • The finding that matters most — there is no failover. Verified from live jobs.json: all 18 enabled agent-mode jobs are pinned to custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable pin, custom:omni (CT600, omniroute:20128), sits on e530c89f2be0 — which is no_agent: true; d40085da631c is disabled. Completing runs inside the window by provider: 210 no_agent / no provider + 1 no_agent on custom:omni. So zero model calls completed in 34 hours, and the fleet has no job that could even report the second node's health.
  • Correction made during this audit. A first pass read Evening Stand-Down Briefing's clean 18:45 runs (both days) as evidence CT600 stayed healthy. Wrong: it is no_agent and never calls a model. Recorded as a standing rule — never read a no_agent job's green row as provider-health evidence — in the incident page and in systems/scheduled-tasks.
  • User-visible cost: silence. Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The weekly Perth New Restaurant was missed outright (next slot 10-22).
  • Recovery, verified live (not assumed). 1b31187397a5 completed 10-15 22:01:56 (streak 4 → 0); 96eb95ce12f2 10-16 00:01:36; b63ed5817212 10-16 00:02:29; ea79a2107e36 10-16 00:09:48. Probes 2026-10-16 00:1x: GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19.
  • Fleet bookkeeping. Non-clean last_status: 3 → 12 (9 new: 08eb86f02d37, ecc1fc13655a, dca8482e4f76, c8d52904e1d6, 06a0a1c8704f, 85edf6772bca, 0e519874d280, 1a8e90ea2271, cbfe434a3b5e; three cleared in the window: b63ed5817212, 1b31187397a5, ea79a2107e36). Open incidents 3 → 12 (9 alerted / 3 detected). Long-standing and pre-storm: 1b23c042e17d (Fri 08:00), 0e753e9720fd (1st 09:00), 066bb5191280 (delivery_failed, dead Telegram thread 34795).
  • Mnemosyne. Both monitors sit on the failing node, so no coverage figures were refreshed for ~34 h; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected.
  • Self-healing gap left alone deliberately. The 10-15 03:00 stocktake failure left raw/stocktake/freshness-20261015.txt unwritten and systems/skills-index.md stamped 10-14. Its next slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the scheduled run.
  • Sessions. 36 sessions across the two days, all cron, zero interactive — fourth consecutive window with no interactive session.

Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new, 9,303 B); systems/scheduled-tasks.md (frontmatter, Count current-state bullet, new Re-verified block); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + blast-radius note on the two monitors); current-state.md (frontmatter + new top section, Updated: line); index.md (Last updated, new incidents entry, scheduled-tasks bullet); log.md (this entry). Nothing removed or archived — no retired job, no obsolete system, so index.md structure gains one page.

Durable fix still open (needs Anthony's call): the single-node topology. All 18 model-calling jobs on one OmniRoute node means this recurs on the next omniroute-1 blip — as it did on 2026-10-09. Options are a second provider pin on a subset of jobs, or router-side failover. Not applied — that is a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs and job/incident IDs only.

[2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead

Window: cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran 2026-10-18 00:31 → 00:5x AWST. Sessions reviewed: 30, all cron, zero interactive — fifth consecutive window with no interactive session, so every finding below is from live system state, not conversation.

Summary

  • omniroute-1 (custom:omni-101) has a second failure mode: 503 load-shedding. 9 model calls across 7 jobs died on RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. between 10-16 08:00:37 and 10-17 03:00:28 (~19 h). The control plane stayed up the whole time (/api/health 200 {"status":"ok"}, /v1/models 401 — re-probed 10-18 00:33), so a green health endpoint is not evidence that the model path is serving. This is not a repeat of the 10-14 storm: that was reachability (Connection error., Errno 111), this was the router serving and refusing under load. → incidents/2026-10-16-omniroute-503-resource-pressure (new).
  • The burst ended on its own and the fleet is green again. 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01), 10-18 so far 7 / 0; no failure row of any kind since 10-17 03:00. Every affected job re-ran clean on its next slot — b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31, ecc1fc13655a + ea79a2107e36 both 10-17 12:02, each auto-closing its own incident. No change was made on the node; nothing was restarted or repinned.
  • Mosquito Tracker removed — fleet 29 → 28 (1b31187397a5). Absent from jobs.json (updated_at 2026-10-18T00:36:17+08:00), last execution 10-16 15:00 (completed, delivered). The three disabled jobs are unchanged, so 25 enabled / 3 disabled. Non-clean last_status 12 → 8; only 1b23c042e17d (error, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
  • The watchdog double-counts. 96eb95ce12f2 exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one sentinel. Read failures per job, not per row.
  • Second consecutive stocktake hole, again left alone deliberately. The 10-17 03:00 slot failed, so raw/stocktake/freshness-20261017.txt is missing and systems/skills-index.md is stamped 10-16; 10-15 is missing too (storm). The 10-18 03:00 slot closes it — no manual regeneration, because a hand-run would race the scheduled one.
  • GitHub Trending Radar is dead at the delivery end. The Telegram bot token in its config.json returns HTTP 401 on getMe (probe 10-18 00:35); the job's GitHub PAT still returns 200, so only the delivery credential is gone. The cron row stays green (last_status from the 10-12 09:00 run, last_run_at 2026-10-12T09:00:12). Next exercise Mon 2026-10-19 09:00. Not repaired — a new bot token is Anthony's call, and the secret is not written here.
  • Incidents: 9 opened, 6 closed in-window; open 12 → 8 (5 alerted / 3 detected). The ledger files 7 of the 9 503 rows failure_type: unknown despite the explicit message (the two script-exit rows are typed script), so failure_type still cannot be used to triage.
  • Mnemosyne: both monitors rode the burst (ea79a2107e36 10-17 00:00, b63ed5817212 10-16 12:03), both clean on every slot since — including 10-18 00:00 (delivery_outcome: suppressed). No fresh coverage figures exist (they report to Telegram, not the ledger); last explicit read stays 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %), above the 95 % reindex threshold, so no reindex is due.

Pages touched: incidents/2026-10-16-omniroute-503-resource-pressure.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + stocktake row + Known Issues entry); systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter + new Re-verified section); current-state.md (frontmatter + Updated: line + new top section); index.md (Last updated, new incident entry, scheduled-tasks bullet, skills-index note); log.md (this entry). Nothing removed or archived — no page became obsolete in the window.

Bridge housekeeping (verified 2026-10-18): every page this run touched was byte-identical between the authoring tree (/home/hermes/workspace/agent-estate-wiki) and the vault copy (/home/hermes/wiki/agent-estate-wiki, pages nested under wiki/) before the edits, so the seven files were written to the authoring tree first, committed and pushed (4137dd6), then copied into the vault — sha256 matched on all seven. The vault-only pages (wiki/systems/ignis.md, wiki/systems/nextcloud.md) and the vault's own wiki/raw/stocktake/ series were not touched; nothing was deleted and no one-way sync was run. Drift found, not fixed: incidents/2026-09-02-sftpgo-outage.md exists only in the authoring tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.

Durable fix still open (needs Anthony's call): the single-node topology — all 18 model-calling jobs on custom:omni-101 with no failover, now seen failing two different ways in five days. Options remain a second provider pin on a subset of jobs, or router-side failover. Not applied — a routing decision, not an audit one.

No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.

[2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST

Window verdict: one failure all day, and it was a failure mode the wiki had never recorded. Fleet 28 jobs (25 enabled / 3 disabled); 10-18 = 168 completed / 1 failed against 10-17 = 163 / 5; 10-19 so far 5 / 0 plus this audit. 11 sessions in the window, all cron — no interactive sessions, so no user decisions arrived to document; every finding below is machine-observed.

  • A model stream repeated itself and Hermes delivered the placeholder to Telegram. ecc1fc13655a (FreshRSS Midday, 12:00) wrote "Your request was not processed. Send it again if you still want me to carry it out.", delivered it (delivery_outcome: delivered, 12:08:11) and then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response. New page → incidents/2026-10-18-degenerate-stream-canned-substitute. The open question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue) instead of substituting text on the job's normal route?
  • Mattermost: cause found — CT255 is stopped, superseding the 10-11 entry's "cause not determined". Reconnect loop unchanged (attempt 471 at 00:14:07, connect timed out after 30s, 5-minute cadence), peer offline 29d, and the host answers neither TCP :8065 nor ICMP. Platform config still says enabled: true, so the loop will keep logging until the container is started or the block retired.
  • New page: systems/push-notifications — the bridge is broken and has been for as long as the logs go back. status.json: connected: false, error: …connection refused (nothing listens there on this machine); the bridge dials HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net, which /etc/hosts resolves to 127.0.1.1 where nothing serves 443, while the gateway itself is healthy on 0.0.0.0:9119 (/ → 302) and Serve still maps the name to it. 23,469 occurrences of that string across agent.log(.1-3) + errors.log(.1-2). Fix documented, not applied.
  • Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break. working_memory had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18 03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in the page); 1 unresolved conflict. The 24 h threshold in mnemosyne-health-check.sh:92 will keep flagging idle weekends.
  • Stocktake landed and closed the two-day hole: 1a8e90ea2271 exited 0 on 10-18 03:00, freshness-20261018.txt written, systems/skills-index regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (dd53e5f).
  • Carried forward, still open: the GitHub Trending Radar delivery token (getMe → 401, verified 10-18) — its next exercise is Mon 2026-10-19 09:00; the single-node custom:omni-101 topology with no model-call failover; and the vault drift below.

Pages touched: incidents/2026-10-18-degenerate-stream-canned-substitute.md (new); systems/push-notifications.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet + FreshRSS row + totals); systems/messaging-integrations.md (frontmatter + new Mattermost section); systems/mnemosyne.md (frontmatter + new tier/health block); infrastructure/tailscale.md (frontmatter + inventory header + three missing peers + mattermost age); infrastructure/proxmox.md (CT255 age); current-state.md; index.md; log.md. Nothing removed or archived — no page became obsolete and no one-way sync was run.

Bridge housekeeping + vault drift (verified 2026-10-19 00:2x). Of the ten files this run touched, six were byte-identical between the authoring tree and the vault copy before the edits; two more (infrastructure/tailscale.md, infrastructure/proxmox.md) differed only because the vault holds older versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (wiki/ nesting) with sha256 verified per file.

Vault drift measured: against authoring HEAD, the vault holds 74 identical pages, 11 stale and 12 missing. Stale: infrastructure/network-map.md, infrastructure/tailscale.md, infrastructure/proxmox.md, infrastructure/docker-services.md, systems/skills-index.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, incidents/2026-08-01-mnemosyne-capture-failure.md. Missing: incidents/2026-09-02-sftpgo-outage.md (flagged by the 10-18 audit and still absent), the seven agents/* pages, reports/communities-wa-social-media-audit-2026.md, CONTRIBUTING.md, plus this run's two new pages (copied immediately after). The vault repo's own HEAD is still 9bacd8e mirror(2026-10-16) with a working tree carrying uncommitted 10-17/10-18 changes, and it has no remote, so it cannot be pushed. Not fixed — a full mirror pass is a bigger call than an audit's mandate; flagged here.

No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.

[2026-10-20] audit | Nightly wiki audit — omni-101 503 burst 2 (2026-10-19)

  • Incident re-opened. incidents/2026-10-16-omniroute-503-resource-pressure went back to status: active. A second, identical HTTP 503 … resource pressure burst hit omniroute-1 on 2026-10-19 — 9 model calls across 8 jobs (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 (2026-10-16 08:00 → 10-17 03:00). Whole day 159 completed / 13 failed; 2026-10-18 was 168 / 1.
  • Facts corrected from the source of truth (~/.hermes/cron/jobs.json, executions.db) instead of being carried forward: the burst-2 table's job names, schedules and model aliases were wrong (85edf6772bca is Tech & AI Newsletter Digest (Analytical), not "Fringe Festival Plan"; ea79a2107e36 is the Mnemosyne Health Monitor, not a news digest; ecc1fc13655a is FreshRSS Midday News Briefing), 08eb86f02d37 fails at 08:31 (not 08:01, it is a 30 8 * * * job), and 2026-10-17 reads 163 / 5, not 162 / 5. All eight failing jobs read provider: custom:omni-101, model: smart-route — the burst-1 note's wider alias mix (deepseek-paid-smart, free-IA, dark) no longer matches jobs.json and is now flagged in the page for confirmation rather than silently overwritten.
  • Incident ledger read properly for the first time. Unclosed rows = 13 (12 alerted / 1 detected), not the 7 the 10-19 entry recorded — the earlier figures used a narrower filter. A raw closed_at IS NULL count reads 12 at 10-19 00:20 and 13 now; the entry states the query so future entries compare like with like. 4 rows opened and 2 closed on 10-19, plus ea79a2_747b27556fd0 closed 10-20 00:04:24.
  • Pages updated: incidents/2026-10-16-omniroute-503-resource-pressure.md (burst-2 table rebuilt from jobs.json, new findings, status back to active), systems/scheduled-tasks.md (new current-count bullet, non-clean last_status rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected), systems/omniroute.md (new Recent Changes section), current-state.md (new Recent Changes section), index.md (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), log.md.
  • Nothing removed or archived — no page became obsolete and no one-way sync was run.
  • Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 15 sessions in the window, all cron — no interactive sessions, so there were no user-facing decisions to record.
  • No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
  • Vault drift (measured — sha256 per file against /home/hermes/wiki/agent-estate-wiki): 76 identical / 15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512 "missing", but those are build artefacts under public/, package*.json, serve.py and raw/stocktake/, which the vault deliberately does not carry.) All six pages touched by this audit were copied in with per-file sha256 verification. The stale pages left unmirrored — infrastructure/docker-services.md, infrastructure/network-map.md, systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md, systems/obsidian.md, systems/skills-index.md, incidents/2026-08-01-mnemosyne-capture-failure.md — are authoring-tree edits made from other agents' sessions; a full vault mirror pass remains the open call.
  • Not re-run this audit: the public/ Quartz rebuild and wiki-maintain.sh. Authoring commit 95fd7d3, pushed to Gitea Anthony/agent-estate-wiki and confirmed at the remote.