# Wiki Log > Chronological record of wiki actions. Append-only. ## [2026-07-22] init | Fresh Hermes Systems Wiki - Created technical wiki at `/home/hermes/wiki` - Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/ - Wrote SCHEMA.md with 11 operating rules - Wrote index.md as curated Start here dashboard - Reseeded system/infrastructure/runbook stubs for existing homelab - Git initialized, local-only - No secrets, passwords, tokens, or private keys written to vault ## [2026-07-22] audit | External audit pass (Claude, at Anthony's request) - Merged duplicate pages: `systems/omniRoute.md` (empty stub) -> kept `systems/omniroute.md` (populated); repointed 5 links - Merged duplicate pages: `runbooks/safely-update-hermes.md` -> kept as `runbooks/update-hermes-safely.md` (fuller content); repointed 2 links - Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation) - Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin - Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here - Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds) - Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts ## [2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request) - Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory - Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP - Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten - Fixed a broken `review_after: 0001-01-01` in decisions/keep-openrouter-as-fallback.md - New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md) - queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate ## [2026-07-22] correction | Purge Hindsight (per Anthony, direct correction) - Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring - Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md - Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend - Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md - Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use - Confirmed: Mnemosyne is, and has only ever been, this box's memory backend ## [2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony) - Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived - These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki ## [2026-07-25] operations | Tool Search and gateway resilience - Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup - Documented how new skill categories are derived from the first folder below `~/.hermes/skills/` - Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI - Recorded the I/O-controller delegation limitation so an ineffective `IOWeight` is not mistaken for active protection - Added health checks, exact effective settings, backup location, and rollback procedure - No secrets, tokens, or private credentials were written to the wiki ## [2026-07-28] dashboard | Jobs Dashboard feature additions - Fixed bulk status update (payload shape mismatch: `{ids, status}` → `{ids, updates: {status}}`) - Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers) - Added Phone Screen as a dedicated KPI filter card (separated from Interviews) - Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh) - Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion `Notes / JD Summary` - Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API) - Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager - Updated Synergy and Dept of Communities to Phone Screen status - Updated wiki runbook with full dashboard feature documentation and API endpoint table ## [2026-07-27] incident | Job Radar search and dashboard recovery - Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow - Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require `MISSING_COVER_LETTER_COUNT=0` - Recorded the production cron model (`gpt-5.6-terra`) and verified first inherited fallback (`opencode-go/mimo-v2.5`) - Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing - Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both `:443` Serve and `:8443/webhook` Funnel - Added [[runbooks/job-radar]] and updated scheduled-task, Tailscale and index references - Verified no secrets, API keys or private credentials were written to the wiki ## [2026-07-29] update | Steel Browser deployment — browser backend switch - **New page:** `systems/browser-backend.md` — rewritten to reflect Steel Browser as primary, Camoufox as fallback - **New page:** `comparisons/steel-vs-camoufox.md` — comparison of the two backends - **Updated:** `infrastructure/hosts.md` — added CT201 and CT460 to verified hosts table - **Updated:** `infrastructure/docker-services.md` — added Steel Browser compose file - **Updated:** `infrastructure/tailscale.md` — added CT201 Steel serve routes - **Updated:** `index.md` — browser-backend link description - **Updated:** `current-state.md` — added 2026-07-29 changes section - **Config change:** `browser.cdp_url` set to `http://100.96.244.39:9223` in ops profile ## [2026-07-30] update | Open WebUI updated on CT245 - Updated image to `ghcr.io/open-webui/open-webui:main` (new digest e97bf9531916, Jul 27) - Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image - Restarted container after OOM hang - Updated `infrastructure/proxmox.md` ## [2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed - Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap - Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded - Ran `mnemosyne reindex --yes` to rebuild all embeddings — working memory now at 100% - Updated health check script (`mnemosyne-health-check.sh`) to report actual embedding coverage via new helper script (`check_embedding_coverage.py`) - Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95% - Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed - DB grew from 8.7M to 21M with the new embeddings ## [2026-08-01] incident | Mnemosyne capture pipeline repaired - Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no `working_memory` row existed. - Corrected stringified structured YAML (`sync_roles`, `skip_contexts`, `reflect`, `sleep_threshold`, `ignore_patterns`) and enabled profile isolation; `hermes config check` passed. - Verified the repair with a second CLI marker that persisted as an exact normalized `[USER]` database row. - Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route `:9119`; added authoritative identity/preference/workflow slots. - Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained. - Added [[incidents/2026-08-01-mnemosyne-capture-failure]] and refreshed [[systems/mnemosyne]] plus [[systems/memory-backend]]. ## [2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation - Rewrote [[comparisons/steel-vs-camoufox]] — full Steel experiment story, timeline, lessons learned - Rewrote [[systems/browser-backend]] — Camoufox primary, Steel disabled - Updated [[current-state]] — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename - Updated [[concepts/browser-automation]] — added Steel experiment lesson - Created [[decisions/rollback-steel-to-camofox]] — formal decision record - Updated [[systems/scheduled-tasks]] — job count 25, Uber Eats upgrade, GH Trending Daily removed - Updated [[systems/mnemosyne]] — vector embedding coverage gap documented - Updated [[systems/model-providers]] — DeepSeek V4 Flash 0731 details - Created [[reports/uber-eats/index]] — landing page for monthly spend reports - Updated [[decisions/index]] — linked new decision page - Updated [[index.md]] — browser description, reports section, last-updated date - Created nightly wiki audit cron job (23:59 AWST) ## [2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync - Ops profile (`~/.hermes/profiles/ops/config.yaml`): enabled `mnemosyne` plugin with `profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false` - Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination - Verified end-to-end: memory writes land correctly in both profiles independently - Updated [[systems/mnemosyne]] — profile isolation section now documents ops config - Updated [[current-state]] — added ops profile Mnemosyne entry - Updated [[runbooks/multiplexer-setup]] — fixed "Elephant ops" → "Tusk ops" in architecture diagram - Fixed [[systems/memory-backend]] — "Shared surface empty" section renamed to reflect intentional isolation decision ## [2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation - Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances - Reworked `deepseek-flash-free` and `good-free` on CT600: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort - Created [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] - Updated [[current-state]] — added OmniRoute free-DeepSeek mitigation to Recent Changes - Updated [[systems/omniroute]] — added Recent Changes (2026-08-04) section - Updated [[systems/model-providers]] — added mitigation entry under Notable Model Updates, updated `updated` date ## [2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite - **OmniRoute three-layer fix documented:** npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. `is_active=0` does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-07) section with full root cause analysis - **Updated [[systems/nanobot]]** — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on. - **Updated [[current-state]]** — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump - **Rewrote [[systems/scheduled-tasks]]** — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28. - **Wiki audit cron error noted:** model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job - **Updated [[index.md]]** — last-updated date 2026-08-04 → 2026-08-07 ## [2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review - **Portainer password reset documented:** CT245 admin user confirmed as `anthony` (not `admin`). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to [[current-state]]. - **SFTPGo security review documented:** CT299 findings — root access, unpinned fingerprint, `//` root path, F1 quota mystery. Added to [[current-state]] and [[infrastructure/proxmox]]. ## [2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review - **Cron job model repinning:** 13 jobs repinned from dead/stale provider combos. 3 from `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker from `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named jobs (e.g. `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`) → `smart-route`. Daily Marketing Job Radar → `gpt-5.6-terra`. - **Hermes UI vite config revert:** Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460. - **Phase One estate review completed:** Manual cron run (ID `a9359b7db27b`) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, CT333 Nanobot has live `jobs.json` with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes. - **Updated [[current-state]]** — added 2026-08-08 changes section - **Updated [[systems/scheduled-tasks]]** — Phase One review marked completed, model repinning noted in Known Issues ## [2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue - **Cron job failures documented:** Four jobs failed on 2026-08-10 with two error patterns: - TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest - RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup - **Root cause identified:** Disk at 83% (11GB free on 65GB), `state.db` at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues. - **Updated [[current-state]]** — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors - **Updated [[systems/scheduled-tasks]]** — added session storage failures to Known Issues, updated verified_on to 2026-08-10 - **OpenCode WebSocket proxy errors noted:** Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient. - **No new pages created** — all changes applied to existing pages ## [2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals - **OmniRoute AutoRoute variants documented:** `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: `auto/best-free` routed to Antigravity. Updated [[systems/omniroute]]. - **OpenViking memory provider evaluated, NOT adopted:** single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains `memory.provider: mnemosyne`. Added to [[comparisons/memory-backends]]. - **FreeLLMAPI router internals documented:** 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default `freellmapi-auto`. Added to [[comparisons/model-routers]]. - **No new pages created** — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents. ## [2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration - **OmniRoute migrated from CT600 to CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. All facts verified live (`pct list`, `grep`, Tailscale DNS). - **Updated [[systems/omniroute]]** — Current Configuration rewritten for `omni`/CT601, added Recent Changes (2026-08-13), bumped `updated`/`verified_on` to 2026-08-13. - **Updated [[infrastructure/proxmox]]** — CT600 marked stopped/retired, CT601 added running; bumped `updated`. - **Updated [[current-state]]** — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix. - **Updated [[infrastructure/hosts]]** — OmniRoute proxy entry → `omni`; CT600 reference marked verified/retired. - **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13. - No new pages created; all changes applied to existing pages. No secrets written. ## [2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated - **Skills index drift incident:** `systems/skills-index.md` claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating `regenerate-skills-index.sh` script, and adding cron job `1a8e90ea2271` (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See [[incidents/2026-08-15-skills-index-drift]]. - **Updated [[systems/skills-index]]** — added auto-regeneration notice, updated counts (66→330), bumped `updated`/`verified_on` to 2026-08-15. - **Updated [[systems/scheduled-tasks]]** — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates. - **Updated [[systems/hermes-agent]]** — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates. - **Updated [[index.md]]** — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15. - **Created [[incidents/2026-08-15-skills-index-drift]]** — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron). - No secrets written. ## [2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash - **New page:** `systems/nextcloud.md` — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages. - **Updated [[current-state]]** — added Recent Changes (2026-08-17) section with ncdownloader fix details. - **Updated [[infrastructure/proxmox]]** — CT270 entry now shows Nextcloud version and ncdownloader status. - **Updated [[index.md]]** — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17. - No secrets written. ## [2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk - **OmniRoute upstream capacity pressure documented:** `ServiceUnavailableError: Structurally heavy chat request capacity is busy` reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded. - **Agent Zero sluggishness confirmed as orchestration overhead:** GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026). - **Hermes model verification quirk documented:** Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config `http://omni:20128/v1` is correct — `/v1` is required. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-16) section, bumped `updated`/`verified_on` to 2026-08-16. - **Updated [[current-state]]** — added Recent Changes (2026-08-16) section, bumped `updated` to 2026-08-16. - **Updated [[index.md]]** — bumped last-updated date to 2026-08-16. - No new pages created. No secrets written. ## [2026-08-18] incident | Mnemosyne capture pipeline broken (active) - **New page:** `incidents/2026-08-18-mnemosyne-capture-pipeline` — working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification. - **Updated [[systems/mnemosyne]]** — added 2026-08-18 incident to History and Evidence, bumped `updated`/`verified_on` to 2026-08-18, added cross-reference to Related. - **Updated [[current-state]]** — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped `updated` to 2026-08-18. - **Updated [[index.md]]** — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18. - **Marked [[incidents/2026-08-01-mnemosyne-capture-failure]]** — status changed from `active` to `resolved` (that earlier incident was fixed). - No secrets written. ## [2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history - **Cron job failures documented (Weekly Review session):** Three distinct failure patterns identified in past 7 days: - `wiki-stocktake-maintenance` (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from `cx/gpt-5.6-luna` → `go-cheap`/`opencode-go/gpt-5.6-luna`, job unpinned. Requires explicit provider/model pinning. - CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected. - Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping. - **Updated [[systems/scheduled-tasks]]** — added Known Issues section with cron failure patterns, bumped `updated`/`verified_on` to 2026-08-23. - **Updated [[systems/mnemosyne]]** — added Mnemosyne Watchdog Aug 19 incident to History section, bumped `updated`/`verified_on` to 2026-08-23. - **Updated [[current-state]]** — added Recent Changes (2026-08-23) section with cron failures summary. - **Updated [[index.md]]** — bumped last-updated date to 2026-08-23. - No secrets written. ## [2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status - **New page:** `systems/ignis.md` — documented Ignis browser-native Obsidian (Docker `ignis-subpath:0.8.9`, port 8080, Tailscale Serve at `hermes.kangaroo-eel.ts.net/ignis`). Volume mount: `/home/hermes/wiki/agent-estate-wiki:/vaults/wiki`. Verified live: container running, Tailscale route active. - **Updated [[systems/model-providers]]** — added `command-code` provider (`https://api.commandcode.ai/provider/v1`) to Current Provider Stack. Bumped `updated`/`verified_on` to 2026-08-19. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-19) section documenting Command Code provider, `deepseek-flash-mix` combo, and Tavili search proxy status. Bumped `updated`/`verified_on` to 2026-08-19. - **Updated [[infrastructure/docker-services]]** — added Ignis compose file and service definition. - **Updated [[infrastructure/hosts]]** — added Ignis to CT460 row. - **Updated [[systems/obsidian]]** — added Ignis as browser interface, corrected vault path to `agent-estate-wiki/`, added Quartz static site reference. - **Updated [[current-state]]** — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status. - **Updated [[index.md]]** — added Ignis link to Core systems, bumped last-updated to 2026-08-19. - No secrets written. ## [2026-08-26] update | Nightly wiki audit — Cron job reconciliation - **Updated [[systems/scheduled-tasks]]** — reconciled live `hermes cron list` (27 jobs) against wiki table (29 entries). Removed: `Workday start reminder` (no longer in live list), `CT 221 Miner Watchdog` (job `d5516d0a5147` removed; CT221 status unknown), `Review Agent Estate Phase One` (one-time job completed 2026-08-08). Added: `weekly-save-restricted-bot-restart` (`0 3 * * 0`, local, no-agent). Updated count from 29 to 27. - **Updated [[current-state]]** — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence. - **Updated [[index.md]]** — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27). - **Known Issues cleanup** — removed stale CT221 miner watchdog SSH failure entry (job no longer exists). - No secrets written. ## [2026-09-02] incident | SFTPGo outage on CT299 - **New page:** `incidents/2026-09-02-sftpgo-outage.md` — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), `tailscaled` died; PVE `pct start` failed due to 5 stale `lxc-attach` processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, `pct stop 299 --skiplock` + `pct start 299`. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. - **Updated [[current-state]]** — added Recent Changes (2026-09-02) section, bumped `updated`/`verified_on` to 2026-09-04. - **Updated [[infrastructure/proxmox]]** — CT299 entry already documented incident; no change needed. - No secrets written. ## [2026-08-28] update | Department of Communities role and social-media audit - **New page:** `reports/communities-wa-social-media-audit-2026` — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role. - **Updated [[systems/current-profile]]** — recorded current employment context and linked the report. - **Updated [[index.md]]** — added the report to the Reports section and refreshed the last-updated date. - No secrets written. ## [2026-09-20] update | Nightly wiki audit — No changes in past 24h - **No new pages created** — no new systems, decisions, or incidents in the past 24 hours. - **No existing pages updated** — all wiki content remains current. - **Freshness audit run:** `freshness-20260920.txt` generated at 03:01 AWST (scheduled cron). - **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`). - **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`. - No secrets written. ## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation - **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`. - **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`. - **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30. - **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section. - **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps. - **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup). - **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed. - No secrets written. ## [2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene - **No new pages created** — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages. - **Updated `systems/scheduled-tasks.md`** — table and counts re-verified against live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, ×5), `1a8e90ea2271` (HTTP 410, model retired upstream, ×7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: `1b23c042e17d` and `c8d52904e1d6` delivery targets now `telegram:-1004321904721:102`; `c0a9fa314542` flagged for a revoked Telegram bot token; bare-`telegram` targets noted on `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`. Known Issues restructured into Resolved / Open / Historical. - **Updated `systems/messaging-integrations.md`** — new **Delivery target semantics** section: a bare `deliver: "telegram"` resolves to the home DM (`1793951355`), *not* to the primary group, so a group topic ID attached to it hard-fails with `Message thread not found` and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs. - **Updated `systems/model-providers.md`** — provider stack rebuilt from live `config.yaml`: 5 custom providers (`omni-101` 5,384 models, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46); `custom:omni` recorded as removed. Default model corrected `resilient-free` → `smart-route`. New **Retired / Unavailable Models** table (retired `qwen3.5:397b`, plan-gated `gpt-5.6-terra`, catalog-missing `deepseek-v4-flash`) with the lesson that a raw model pin never self-recovers. - **Updated `systems/mnemosyne.md`** — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as `[USER] [Anthony Martin]` memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy. - **Updated `systems/hermes-agent.md`** — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins. - **Updated `index.md` and `current-state.md`** — last-updated bumped to 2026-10-01 with a Recent Changes block. - **Vault sync:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under `wiki/`, not at the tree root, and `index.md` sits at the vault root. - No secrets written. ## [2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split - **New page:** `incidents/2026-10-01-cliproxy-connection-auth-expired.md` — job `06a0a1c8704f` (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with `HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Documented as its own incident because it is a **credential** fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (`1b23c042e17d`, `79f9bb41ecf5`). - **Updated `systems/scheduled-tasks.md`** — counts re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **3 → 6**. New on 2026-10-01: `06a0a1c8704f` (HTTP 401), `0e753e9720fd` (HTTP 400 — same NVIDIA model pin as the already-failing `fad5a86979d4`, confirming the shared-pin diagnosis), `066bb5191280` (delivery failed; report itself generated). Streaks advanced: `dca8482e4f76` ×5→×6, `1a8e90ea2271` ×7→×8, `fad5a86979d4` ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees. - **Updated `systems/messaging-integrations.md`** — corrected a wrong page fact. The page previously stated that a bad `thread_id` always loses the message. The 2026-10-01 records show **two variants**: `Message thread not found` (hard failure, message lost — the 2026-09-30 case) and `configured thread_id for telegram: was not found; delivered without thread_id` (a thread-less fallback send *does* occur, yet the run is still recorded `delivery_outcome: failed`). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-`telegram` jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today". - **Updated `systems/current-profile.md`** — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no `~/.hermes/profiles/default/` directory (the default profile is the top-level config; `ops` is the only named sub-profile), the model default is `smart-route` (not `freellmapi-auto`), the default provider is `custom` → `http://100.78.63.20:20128` / OmniRoute (litellm is now only the last fallback entry), and the config is `_config_version: 46`. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings. - **Updated `current-state.md`** — new **Recent Changes (2026-10-02)** block; frontmatter dates bumped. - **Updated `index.md`** — last-updated to 2026-10-02; cron line now notes the failing count. - **Structural finding — the wiki exists as two trees and they have drifted both ways.** `/home/hermes/workspace/agent-estate-wiki` is canonical: it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build and is what `wiki-maintain.sh` commits. `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault copy the iPad reads (pages under `wiki/`, no git remote). The audit prompt names only the vault path. The trees have also lost pages in *opposite* directions — `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist only in the vault, while `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` and `reports/communities-wa-social-media-audit-2026.md` exist only in canonical — so canonical `index.md` currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to **both** trees. - No secrets written; all credential-shaped values in config remain redacted. ## [2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount - **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), failing set still **6** but the composition moved. `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` that the cliprox incident predicted — making it **2 of the 3** `cliprox/terra` jobs confirmed broken, which retires any "one-job fluke" reading. `79f9bb41ecf5` is the last untested job on that pin (next due 2026-10-04 18:00). **Streak recount** against `executions.db`: `dca8482e4f76` ×6→**×7**, but `1a8e90ea2271` ×8→**×7** — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). **Resolved:** `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). **Downgraded:** the GitHub Trending Radar revoked-token claim is now "unverified at last run" — `c0a9fa314542`'s 2026-09-30 19:22 run completed *and delivered*. New Known Issue recording that **a delivered notice is not a successful job** (both cliprox failures read `delivery_outcome: delivered` on a `failed` status). - **New page `incidents/2026-08-26-work-outlook-basic-auth-disabled.md`** — the `work` Outlook mailbox has been unreachable since **2026-08-26**: himalaya is configured for password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. TCP/TLS to `outlook.office365.com:993` is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap. - **Updated `systems/messaging-integrations.md`** — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, `work` unreachable; frontmatter `updated`/`verified_on` bumped. Added the tooling consequence: `email/mail-lookup` resolves the account before searching, so work-mail requests hit this wall and must be reported as *account unreachable*, not "no results found". - **Updated `systems/hermes-agent.md`** — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to **six jobs across three model faults plus one credential fault**, and a second entry added for the stock-take job being down; the `Email: connected` bullet corrected. - **Updated `current-state.md`** and **`index.md`** — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped. - **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root. - No secrets written; all credential-shaped values in config remain redacted. ## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST. **New pages** - `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). - `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms). **Updated** - `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha. - `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented. - `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added. - `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation. - `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live. - `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced. - `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted. - `infrastructure/hosts.md` — CT450 row updated to v0.21.5. - `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks). - `runbooks/provider-health.md` — stale v0.19.0 version reference corrected. - `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated. **Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`. ## [2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7 Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST. - **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json` + `hermes cron list`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **6 → 7**. The one new failure event in the window is `79f9bb41ecf5` (**Weekly Review**, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. It was the last untested job on the `cliprox/terra` pin, so that pin is now **3 of 3 confirmed broken** — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`) and the Known Issues Open list all updated; frontmatter dates bumped. - **Updated `incidents/2026-10-01-cliproxy-connection-auth-expired.md`** — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still **Open** — needs the OmniRoute connection re-authenticated before any repin. - **Updated `current-state.md`** — new **Recent Changes (2026-10-05)** block: the cliprox 3/3 settlement, `1a8e90ea2271`'s repeat 410 on 2026-10-04, confirmation that **no interactive sessions occurred in the window** (all 10 active sessions were cron), and a streak-counting caveat. - **Updated `index.md`** — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed". - **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`. - No secrets written; all credential-shaped values remain referenced by connection ID only. ## [2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0 Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST. **New** - `incidents/2026-10-05-dead-model-pin-repin.md` — the seven-job dead-model-pin repin. Records the four dead pins (`cliprox/terra` HTTP 401 ×3, `ollama-cloud/qwen3.5:397b` HTTP 410 ×1, `nvidia/deepseek-ai/deepseek-v4-flash` HTTP 400 ×2, `command-code/gpt-5.6-terra` HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (`free-stack` ×6, `smart-route` ×1 on `custom:omni-101`) over another raw model name. **Updated** - `systems/scheduled-tasks.md` — Count block rewritten: **0 failing** (was 7), with all 30 `failure_streak` counters at 0; four jobs carry a stale pre-repin `last_status: error` that clears on their next scheduled run. Seven job-table rows updated (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`, `dca8482e4f76`), plus `c0a9fa314542`. New `### Resolved 2026-10-05` subsection in Known Issues; five resolved bullets removed from `### Open`, leaving bare-`telegram` targets, the two-wiki-tree drift, the work-mailbox block, the stale `custom:omni` key and the delivered-notice rule as the standing open set. - `index.md` — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed. - `current-state.md` — new Recent Changes (2026-10-06) block; frontmatter dates bumped. **Findings worth the ink** - **`dca8482e4f76` (Daily Marketing Job Radar) delivered to topic 1030 at 23:56** — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for `79f9bb41ecf5` (23:39) and `06a0a1c8704f` (23:45). - **GitHub Trending Radar's revoked-token diagnosis is dead.** `c0a9fa314542` ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No `getMe` → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright. - **`delivery_outcome: delivered` still does not mean the job worked.** Every pre-repin failure recorded `delivered`, because the failure notice is itself the deliverable. Judge on the execution `status` column. - **A stale notice opened the review.** Anthony was prompted by a `85edf6772bca` notice that had sat in topic `346` since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** - **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` and `dca8482e4f76` at 23:33/23:48, superseded by their successful re-runs) and 1 `running` (this audit). **Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by connection ID only. ## [2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST. **Updated** - `systems/scheduled-tasks.md` — Count block corrected: the stale pre-repin `last_status: error` set is now **three** (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`), not four — `1a8e90ea2271` cleared its own when it ran at 03:00. Added a **Re-verified 2026-10-07 AWST** bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 `completed` / 0 `failed` / 1 `running`; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended. - `index.md` — last-updated → 2026-10-07; cron-fleet line re-dated (**0 failing as of 2026-10-07**, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration. - `current-state.md` — new Recent Changes (2026-10-07) block; frontmatter dates bumped. **Findings worth the ink** - **`1a8e90ea2271` ran itself at 03:00 and proved the repin.** `wiki-stocktake-maintenance` completed **2026-10-06 03:00:31 → 03:03:32** on `custom:omni-101` / `free-stack` — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired `qwen3.5:397b`). It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt` and committed to Gitea as `33f4e95`. Its four HTTP-410 incidents (`6df957a05ec6`, `d284168d3663`, `cda67de5b3e3`, `e31584909bd2`) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. **The daily manual `wiki-maintain.sh` run can stop.** - **Vault mirror drift found and partly repaired.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item. - **Window tally** (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00): 176 `completed`, 0 `failed`, 1 `running` (this audit) — 177 rows. 14 sessions, all `cron`; no interactive (non-cron) sessions, and no new incident opened. **Vault mirror:** `systems/scheduled-tasks.md`, `systems/skills-index.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by connection ID only. ## [2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST. **Updated** - `systems/scheduled-tasks.md` — new **Re-verified 2026-10-08 AWST** bullet (30 jobs, 27 enabled / 3 disabled, **0 failing**; 176 `completed` / 1 `failed` / 1 `running`; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New **Resolved 2026-10-07** Known Issues section, plus two new **Open** items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs. - `index.md` — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat. - `current-state.md` — new Recent Changes (2026-10-08) block; frontmatter dates bumped. **Findings worth the ink** - **The incident machinery had its first real workout since 2026-09-30 — and it worked.** `62619aa6bcb3` (DHL Tracking) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) was opened 08:00:52, alerted 08:00:58 and **auto-resolved 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed. - **The tracked parcel has arrived.** The 14:00 run reported waybill `8052838382` as `Delivered`, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed. - **Stocktake ran automatically for the second consecutive day** (`1a8e90ea2271`, 2026-10-07 03:00, commit `8c63ef2`; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once. - **Two real defects surfaced from the stocktake's own log.** (1) Step 3 of `wiki-maintain.sh` — the Quartz rebuild — **is skipped on every run**: there is no `quartz.config.ts` and no `npx`, so `public/` is never refreshed and the published site silently ages. (2) `skills-index` **under-reports by one**: 347 `SKILL.md` files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` (the other duplicated basename, `hermes-api-vault`, survives because its two copies sit in different categories). Both are now Open items in [[systems/scheduled-tasks]]. - **Unchanged, still carrying a stale `last_status: error`:** `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-`telegram` targets and the `custom:omni` stale-key references are also unchanged. **Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by job ID and connection ID only. ## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open **Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days. **Pages written** - `incidents/2026-10-09-cron-provider-timeout-storm.md` — new. - `incidents/2026-10-09-memory-recall-relevance.md` — new. - `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped. - `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident. - `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped. - `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added. - `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed. **Findings worth the ink** - **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it. - **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing. - **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`. - **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit. - **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key). **Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by job ID and connection ID only. ## 2026-10-10 — Stocktake (manual run, `wiki-maintain.sh`) - **`systems/skills-index.md` regenerated** — 346 skills / 70 categories (unchanged count; frontmatter `updated`/`verified_on` refreshed to 2026-10-10). Live on-disk count is 347 `SKILL.md`; the one-skill gap is the known de-dup of `productivity/obsidian-git-sync/obsidian-git-sync/`. - **Freshness audit** → `raw/stocktake/freshness-20261010.txt`: **62 stale (≥21d), 23 fresh, 7 no-date of 92 pages**. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No `freshness-20261009.txt` exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands. - **`current-state.md` — new 2026-10-10 section.** The 2026-10-09 timeout storm window re-tallied from `executions.db` (2026-10-09 00:00 → 2026-10-10 00:00 AWST): **163 `completed`, 15 `failed`** — the prior audit recorded 162/15+1 running; the stray `running` row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page). - **DHL monitor re-probed, still enabled.** `62619aa6bcb3` — `enabled: True`, `paused_at: null`, `0 8,11,14,17,20 * * *`, `last_status: ok`. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. **No cron mutation performed by this audit** — retirement stays open item #1. - **`infrastructure/tailscale.md` verified against `tailscale serve status`** — all 13 tailnet Serve routes plus the `:8443/webhook` Funnel match the page. No drift; not edited. - **Not verified / left alone:** the Quartz rebuild step (no `quartz.config.ts`, `npx` cannot fetch `quartz` — `public/` was NOT refreshed, confirming the carried open item); `docker ps` (permission denied on `/var/run/docker.sock` from this session, so `infrastructure/docker-services.md` could not be probed and is left stale); the 80d `2026-07-22` cluster (no live source to check against). - **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices). No secrets written; job/incident IDs only. ## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST) **Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`, 10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5. Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor `62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09 "deleted" claims were false). **Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**: `mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env `795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open. **New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit `paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the `/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault. **Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined. **Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/mnemosyne.md`, `systems/messaging-integrations.md`, `incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-11] stocktake | Daily maintenance + live drift corrections - Ran `wiki-maintain.sh`: skills-index regenerated (346 skills / 70 categories), freshness audit written to `raw/stocktake/freshness-20261011.txt` (62 stale / 24 fresh / 7 no-date of 93 pages) - **OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed.** Verified via `pct list`, `pct config`, `ss` and `tailscale status`: CT600 `omniroute` (100.88.81.19) is RUNNING and serving `:20128`; CT601 `omni` (100.93.204.4) is STOPPED with `onboot=0`. Hermes resolves `custom:omni` → `omniroute:20128` → CT600, not CT601. systems/omniroute.md rewritten with the live node table. - **Third OmniRoute node found:** `omniroute-1` at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider `omni-101` (5384 models, default `smart-route`, carries `skills_hub`). Not in `pct list` — identity unconfirmed. - **Ignis duplicate found:** a second `ignis-ignis-1` container runs on CT200 (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`), separate from CT460's `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them. - **Tailscale route table corrected** against live `tailscale serve status`: `/agent-estate` target is bare `http://127.0.0.1:8769` (not `.../8769/agent-estate-role-dashboard.html`); previously-missing tailnet-only `:9443` → `127.0.0.1:5173` documented. CT201 routes refreshed (8085 targets + `/steel` 5173 + `/steel-api` 3000). - **Proxmox inventory rebuilt** from live `pct list` + `qm list`: added CT475 `agentzero` and VM110 `omarchy`; corrected 282 `ubuntu-server-xfce` to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed rather than guessed. - **Nanobot verified** on CT333: gateway `nanobot-gateway` (PID 2259048) listening on 8765/8766; NOT a systemd `nanobot.service`/`gateway.service` (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified. - **Docker services page corrected:** the old "Docker runtime not available" note was a CT460 socket-permission artifact (`/var/run/docker.sock` is `root:docker srw-rw----`, `hermes` user not in group `docker`, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo). - **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping. - Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d. - Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten. ## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST) **Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`, 10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed / 0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). **Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and [[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %. **New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the 2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; `-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]] (`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point). **Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open, and the skills-index blurb no longer claims no regeneration since 2026-10-07. **Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching (`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill. **Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`, `queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`, `current-state.md`, `index.md`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections **Script run.** `wiki-maintain.sh` completed all three stages: skills-index regenerated from the live skill tree (**346 skills** across 70 categories, unchanged count), freshness audit written to `raw/stocktake/freshness-20261012.txt` (**54 stale / 32 fresh / 7 no-date of 93 pages**), and the Quartz step took its documented no-`quartz.config.ts` path (build skipped, `public/` unchanged — expected, not a failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and written its artifacts, so results were read from the report rather than assumed. **Drift corrected from live probes (2 pages):** 1. **[[systems/nanobot]]** — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on CT333 is **back up**, HTTP `200` on `http://100.122.189.81:8766/`. Port `8900` — the endpoint the disabled OmniRoute `nanobot` provider connection uses — is **still closed**, so `status` stays `degraded` rather than flipping to active. SSH to CT333 is unreachable from CT460, so `whatsapp-bridge.service` was not re-verified. 2. **[[systems/home-assistant]]** — carried `confidence: low` plus a page-wide "cannot confirm HA runs" caveat from 2026-07-22. Live: HTTP `200` on `http://100.68.139.5:8123/`, Tailscale node `homeassistant` active and offering an exit node, VMID 100 running per `qm list`. Confidence raised low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460). **Verified and left stale.** [[systems/openrouter]] and [[systems/nextcloud]] were probed shallowly and nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork. **Pages touched:** `systems/nanobot.md`, `systems/home-assistant.md`, `systems/skills-index.md`, `raw/stocktake/freshness-20261012.txt`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes **Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from `~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former compaction of this job left part of the review written but the log entry and cross-page updates unfinished — completed here. **Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in [[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised the same call that failed**, and green cron status is not delivery proof. (Credential was present; value neither recorded nor repeated.) **Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`, 383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs` and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at **07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis: **workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired. **Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host (CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described the old port model; both pages now match the probe. **Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started 23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to 2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3** (`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 — the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50. **Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at **03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and `dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]] updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry their rows for up to a week after the fault is gone. **Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic **1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned; 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]]. **Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 categories — the known 347-vs-346 on-disk off-by-one persists). **Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag), `systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new `## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite, history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md` (status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only. **Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09** (which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows = 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads `126 / 1`, partial). **Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`, `alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09 timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00 run. → [[incidents/2026-10-05-dead-model-pin-repin]] **Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills / 70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh / 7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit. Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a perfect script run leaves nothing behind when the LLM turn dies. **Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar, Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday, Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01. **Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic 1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping rather than requiring the literal token as a strict control. → [[systems/mnemosyne]] **Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive session. **Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared `fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3** (1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows). **Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list, `## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md` (new Recent Changes + date), `index.md` (date + two blurbs), `log.md`. **Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`, 2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md` (**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt` (**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again (no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item. **Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0 (kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a hypervisor. Live guests recorded from a single `pct list` / `qm list` pass. **Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**, yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak` and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward. **Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live `tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`, `omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`, `omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too (`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified **exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that table needed no change. **Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service` now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with 16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the fault, as diagnosed last cycle. **Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282 `ubuntu-server-xfce` is now **running**, not stopped as recorded. **Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed / 1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23 completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06` (detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled / 3 disabled**. **Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter), `current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt` (new), `log.md`. **Not verified, left as stale:** the remaining 46 stale pages were not probed against a live source, so they stay flagged rather than rewritten. Not re-verified this pass and still single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]], [[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story. No secrets written; host names, IPs, VMIDs and job/incident IDs only. --- ## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit (`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered. **Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).** - **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in `config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the rotations. - **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight). Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15). - **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18 enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`), sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model calls completed in 34 hours**, and the fleet has no job that could even report the second node's health. - **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs (both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident page and in [[systems/scheduled-tasks]]. - **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth New Restaurant was missed outright** (next slot 10-22). - **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0); `96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**. Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. - **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`, `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`). **Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d` (Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795). - **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected. - **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the scheduled run. - **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive window with no interactive session. **Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B); `systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block); `systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter + blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line); `index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry). **Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page. **Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing decision, not an audit one. No secrets written; host names, IPs, VMIDs and job/incident IDs only. ## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead **Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran 2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth consecutive window with no interactive session, so every finding below is from live system state, not conversation. ### Summary - **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across 7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm: that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new). - **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`, `10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since 10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own incident. **No change was made on the node**; nothing was restarted or repinned. - **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json` (`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only `1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck. - **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one sentinel. Read failures per job, not per row. - **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so `raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16; 10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a hand-run would race the scheduled one. - **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json` returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run, `last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot token is Anthony's call, and the secret is not written here. - **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit rows are typed `script`), so `failure_type` still cannot be used to triage. - **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03), both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex threshold, so no reindex is due. **Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry); `systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter + new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section); `index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note); `log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window. **Bridge housekeeping (verified 2026-10-18):** every page this run touched was byte-identical between the authoring tree (`/home/hermes/workspace/agent-estate-wiki`) and the vault copy (`/home/hermes/wiki/agent-estate-wiki`, pages nested under `wiki/`) *before* the edits, so the seven files were written to the authoring tree first, committed and pushed (`4137dd6`), then copied into the vault — sha256 matched on all seven. The vault-only pages (`wiki/systems/ignis.md`, `wiki/systems/nextcloud.md`) and the vault's own `wiki/raw/stocktake/` series were not touched; nothing was deleted and no one-way sync was run. **Drift found, not fixed:** `incidents/2026-09-02-sftpgo-outage.md` exists only in the authoring tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page. **Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on `custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not an audit one. No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only. ## [2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST **Window verdict: one failure all day, and it was a failure mode the wiki had never recorded.** Fleet **28 jobs (25 enabled / 3 disabled)**; `10-18 = 168 completed / 1 failed` against `10-17 = 163 / 5`; `10-19` so far `5 / 0` plus this audit. **11 sessions in the window, all `cron` — no interactive sessions**, so no user decisions arrived to document; every finding below is machine-observed. - **A model stream repeated itself and Hermes delivered the placeholder to Telegram.** `ecc1fc13655a` (FreshRSS Midday, 12:00) wrote *"Your request was not processed. Send it again if you still want me to carry it out."*, **delivered it** (`delivery_outcome: delivered`, 12:08:11) and then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` New page → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The open question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue) instead of substituting text on the job's normal route? - **Mattermost: cause found — CT255 is stopped**, superseding the 10-11 entry's "cause not determined". Reconnect loop unchanged (`attempt 471` at 00:14:07, `connect timed out after 30s`, 5-minute cadence), peer offline **29d**, and the host answers neither TCP `:8065` nor ICMP. Platform config still says `enabled: true`, so the loop will keep logging until the container is started or the block retired. - **New page: [[systems/push-notifications]] — the bridge is broken and has been for as long as the logs go back.** `status.json`: `connected: false`, `error: …connection refused (nothing listens there on this machine)`; the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which `/etc/hosts` resolves to **`127.0.1.1`** where nothing serves 443, while the gateway itself is healthy on `0.0.0.0:9119` (`/` → 302) and Serve still maps the name to it. **23,469** occurrences of that string across `agent.log(.1-3)` + `errors.log(.1-2)`. Fix documented, **not applied**. - **Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.** `working_memory` had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18 03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in the page); 1 unresolved conflict. The 24 h threshold in `mnemosyne-health-check.sh:92` will keep flagging idle weekends. - **Stocktake landed and closed the two-day hole:** `1a8e90ea2271` exited 0 on 10-18 03:00, `freshness-20261018.txt` written, [[systems/skills-index]] regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (`dd53e5f`). - **Carried forward, still open:** the GitHub Trending Radar delivery token (`getMe` → 401, verified 10-18) — its next exercise is **Mon 2026-10-19 09:00**; the single-node `custom:omni-101` topology with no model-call failover; and the vault drift below. **Pages touched:** `incidents/2026-10-18-degenerate-stream-canned-substitute.md` (**new**); `systems/push-notifications.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet + FreshRSS row + totals); `systems/messaging-integrations.md` (frontmatter + new Mattermost section); `systems/mnemosyne.md` (frontmatter + new tier/health block); `infrastructure/tailscale.md` (frontmatter + inventory header + three missing peers + `mattermost` age); `infrastructure/proxmox.md` (CT255 age); `current-state.md`; `index.md`; `log.md`. **Nothing removed or archived** — no page became obsolete and no one-way sync was run. **Bridge housekeeping + vault drift (verified 2026-10-19 00:2x).** Of the ten files this run touched, six were byte-identical between the authoring tree and the vault copy *before* the edits; two more (`infrastructure/tailscale.md`, `infrastructure/proxmox.md`) differed only because the vault holds **older** versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (`wiki/` nesting) with sha256 verified per file. **Vault drift measured:** against authoring `HEAD`, the vault holds **74 identical pages, 11 stale** and **12 missing**. Stale: `infrastructure/network-map.md`, `infrastructure/tailscale.md`, `infrastructure/proxmox.md`, `infrastructure/docker-services.md`, `systems/skills-index.md`, `systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`, `systems/obsidian.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md`. Missing: `incidents/2026-09-02-sftpgo-outage.md` (flagged by the 10-18 audit and still absent), the seven `agents/*` pages, `reports/communities-wa-social-media-audit-2026.md`, `CONTRIBUTING.md`, plus this run's two new pages (copied immediately after). The vault repo's own HEAD is still `9bacd8e mirror(2026-10-16)` with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no remote**, so it cannot be pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here. No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only. ## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19) - **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to `status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 — **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 (2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1. - **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong (`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36` is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**), `08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**, not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1 note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now flagged in the page for confirmation rather than silently overwritten. - **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`), not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL` count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24. - **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from `jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet, non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected), `systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section), `index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`. - **Nothing removed or archived** — no page became obsolete and no one-way sync was run. - **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no interactive sessions, so there were no user-facing decisions to record. - No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only. - **Vault drift (measured — `sha256` per file against `/home/hermes/wiki/agent-estate-wiki`):** 76 identical / 15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512 "missing", but those are build artefacts under `public/`, `package*.json`, `serve.py` and `raw/stocktake/`, which the vault deliberately does not carry.) **All six pages touched by this audit were copied in with per-file `sha256` verification.** The stale pages left unmirrored — `infrastructure/docker-services.md`, `infrastructure/network-map.md`, `systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`, `systems/obsidian.md`, `systems/skills-index.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md` — are authoring-tree edits made from *other* agents' sessions; **a full vault mirror pass remains the open call.** - **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commit `95fd7d3`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote.