Files
agent-estate-wiki/log.md
T

1684 lines
179 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Wiki Log
> Chronological record of wiki actions. Append-only.
## [2026-07-22] init | Fresh Hermes Systems Wiki
- Created technical wiki at `/home/hermes/wiki`
- Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
- Wrote SCHEMA.md with 11 operating rules
- Wrote index.md as curated Start here dashboard
- Reseeded system/infrastructure/runbook stubs for existing homelab
- Git initialized, local-only
- No secrets, passwords, tokens, or private keys written to vault
## [2026-07-22] audit | External audit pass (Claude, at Anthony's request)
- Merged duplicate pages: `systems/omniRoute.md` (empty stub) -> kept `systems/omniroute.md` (populated); repointed 5 links
- Merged duplicate pages: `runbooks/safely-update-hermes.md` -> kept as `runbooks/update-hermes-safely.md` (fuller content); repointed 2 links
- Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
## [2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
- Fixed a broken `review_after: 0001-01-01` in decisions/keep-openrouter-as-fallback.md
- New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
## [2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
## [2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
## [2026-07-25] operations | Tool Search and gateway resilience
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
- Documented how new skill categories are derived from the first folder below `~/.hermes/skills/`
- Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
- Recorded the I/O-controller delegation limitation so an ineffective `IOWeight` is not mistaken for active protection
- Added health checks, exact effective settings, backup location, and rollback procedure
- No secrets, tokens, or private credentials were written to the wiki
## [2026-07-28] dashboard | Jobs Dashboard feature additions
- Fixed bulk status update (payload shape mismatch: `{ids, status}` → `{ids, updates: {status}}`)
- Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion `Notes / JD Summary`
- Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
- Updated Synergy and Dept of Communities to Phone Screen status
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
## [2026-07-27] incident | Job Radar search and dashboard recovery
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require `MISSING_COVER_LETTER_COUNT=0`
- Recorded the production cron model (`gpt-5.6-terra`) and verified first inherited fallback (`opencode-go/mimo-v2.5`)
- Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both `:443` Serve and `:8443/webhook` Funnel
- Added [[runbooks/job-radar]] and updated scheduled-task, Tailscale and index references
- Verified no secrets, API keys or private credentials were written to the wiki
## [2026-07-29] update | Steel Browser deployment — browser backend switch
- **New page:** `systems/browser-backend.md` — rewritten to reflect Steel Browser as primary, Camoufox as fallback
- **New page:** `comparisons/steel-vs-camoufox.md` — comparison of the two backends
- **Updated:** `infrastructure/hosts.md` — added CT201 and CT460 to verified hosts table
- **Updated:** `infrastructure/docker-services.md` — added Steel Browser compose file
- **Updated:** `infrastructure/tailscale.md` — added CT201 Steel serve routes
- **Updated:** `index.md` — browser-backend link description
- **Updated:** `current-state.md` — added 2026-07-29 changes section
- **Config change:** `browser.cdp_url` set to `http://100.96.244.39:9223` in ops profile
## [2026-07-30] update | Open WebUI updated on CT245
- Updated image to `ghcr.io/open-webui/open-webui:main` (new digest e97bf9531916, Jul 27)
- Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
- Restarted container after OOM hang
- Updated `infrastructure/proxmox.md`
## [2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
- Ran `mnemosyne reindex --yes` to rebuild all embeddings — working memory now at 100%
- Updated health check script (`mnemosyne-health-check.sh`) to report actual embedding coverage via new helper script (`check_embedding_coverage.py`)
- Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
- DB grew from 8.7M to 21M with the new embeddings
## [2026-08-01] incident | Mnemosyne capture pipeline repaired
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no `working_memory` row existed.
- Corrected stringified structured YAML (`sync_roles`, `skip_contexts`, `reflect`, `sleep_threshold`, `ignore_patterns`) and enabled profile isolation; `hermes config check` passed.
- Verified the repair with a second CLI marker that persisted as an exact normalized `[USER]` database row.
- Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route `:9119`; added authoritative identity/preference/workflow slots.
- Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
- Added [[incidents/2026-08-01-mnemosyne-capture-failure]] and refreshed [[systems/mnemosyne]] plus [[systems/memory-backend]].
## [2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
- Rewrote [[comparisons/steel-vs-camoufox]] — full Steel experiment story, timeline, lessons learned
- Rewrote [[systems/browser-backend]] — Camoufox primary, Steel disabled
- Updated [[current-state]] — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
- Updated [[concepts/browser-automation]] — added Steel experiment lesson
- Created [[decisions/rollback-steel-to-camofox]] — formal decision record
- Updated [[systems/scheduled-tasks]] — job count 25, Uber Eats upgrade, GH Trending Daily removed
- Updated [[systems/mnemosyne]] — vector embedding coverage gap documented
- Updated [[systems/model-providers]] — DeepSeek V4 Flash 0731 details
- Created [[reports/uber-eats/index]] — landing page for monthly spend reports
- Updated [[decisions/index]] — linked new decision page
- Updated [[index.md]] — browser description, reports section, last-updated date
- Created nightly wiki audit cron job (23:59 AWST)
## [2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
- Ops profile (`~/.hermes/profiles/ops/config.yaml`): enabled `mnemosyne` plugin with `profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
- Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
- Verified end-to-end: memory writes land correctly in both profiles independently
- Updated [[systems/mnemosyne]] — profile isolation section now documents ops config
- Updated [[current-state]] — added ops profile Mnemosyne entry
- Updated [[runbooks/multiplexer-setup]] — fixed "Elephant ops" → "Tusk ops" in architecture diagram
- Fixed [[systems/memory-backend]] — "Shared surface empty" section renamed to reflect intentional isolation decision
## [2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
- Reworked `deepseek-flash-free` and `good-free` on CT600: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
- Created [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
- Updated [[current-state]] — added OmniRoute free-DeepSeek mitigation to Recent Changes
- Updated [[systems/omniroute]] — added Recent Changes (2026-08-04) section
- Updated [[systems/model-providers]] — added mitigation entry under Notable Model Updates, updated `updated` date
## [2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
- **OmniRoute three-layer fix documented:** npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. `is_active=0` does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-07) section with full root cause analysis
- **Updated [[systems/nanobot]]** — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
- **Updated [[current-state]]** — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
- **Rewrote [[systems/scheduled-tasks]]** — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
- **Wiki audit cron error noted:** model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
- **Updated [[index.md]]** — last-updated date 2026-08-04 → 2026-08-07
## [2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
- **Portainer password reset documented:** CT245 admin user confirmed as `anthony` (not `admin`). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to [[current-state]].
- **SFTPGo security review documented:** CT299 findings — root access, unpinned fingerprint, `//` root path, F1 quota mystery. Added to [[current-state]] and [[infrastructure/proxmox]].
## [2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
- **Cron job model repinning:** 13 jobs repinned from dead/stale provider combos. 3 from `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker from `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named jobs (e.g. `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`) → `smart-route`. Daily Marketing Job Radar → `gpt-5.6-terra`.
- **Hermes UI vite config revert:** Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
- **Phase One estate review completed:** Manual cron run (ID `a9359b7db27b`) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, CT333 Nanobot has live `jobs.json` with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
- **Updated [[current-state]]** — added 2026-08-08 changes section
- **Updated [[systems/scheduled-tasks]]** — Phase One review marked completed, model repinning noted in Known Issues
## [2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
- **Cron job failures documented:** Four jobs failed on 2026-08-10 with two error patterns:
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
- **Root cause identified:** Disk at 83% (11GB free on 65GB), `state.db` at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
- **Updated [[current-state]]** — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
- **Updated [[systems/scheduled-tasks]]** — added session storage failures to Known Issues, updated verified_on to 2026-08-10
- **OpenCode WebSocket proxy errors noted:** Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
- **No new pages created** — all changes applied to existing pages
## [2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
- **OmniRoute AutoRoute variants documented:** `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: `auto/best-free` routed to Antigravity. Updated [[systems/omniroute]].
- **OpenViking memory provider evaluated, NOT adopted:** single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains `memory.provider: mnemosyne`. Added to [[comparisons/memory-backends]].
- **FreeLLMAPI router internals documented:** 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default `freellmapi-auto`. Added to [[comparisons/model-routers]].
- **No new pages created** — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.
## [2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
- **OmniRoute migrated from CT600 to CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. All facts verified live (`pct list`, `grep`, Tailscale DNS).
- **Updated [[systems/omniroute]]** — Current Configuration rewritten for `omni`/CT601, added Recent Changes (2026-08-13), bumped `updated`/`verified_on` to 2026-08-13.
- **Updated [[infrastructure/proxmox]]** — CT600 marked stopped/retired, CT601 added running; bumped `updated`.
- **Updated [[current-state]]** — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
- **Updated [[infrastructure/hosts]]** — OmniRoute proxy entry → `omni`; CT600 reference marked verified/retired.
- **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13.
- No new pages created; all changes applied to existing pages. No secrets written.
## [2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
- **Skills index drift incident:** `systems/skills-index.md` claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating `regenerate-skills-index.sh` script, and adding cron job `1a8e90ea2271` (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See [[incidents/2026-08-15-skills-index-drift]].
- **Updated [[systems/skills-index]]** — added auto-regeneration notice, updated counts (66→330), bumped `updated`/`verified_on` to 2026-08-15.
- **Updated [[systems/scheduled-tasks]]** — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
- **Updated [[systems/hermes-agent]]** — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
- **Updated [[index.md]]** — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
- **Created [[incidents/2026-08-15-skills-index-drift]]** — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
- No secrets written.
## [2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
- **New page:** `systems/nextcloud.md` — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
- **Updated [[current-state]]** — added Recent Changes (2026-08-17) section with ncdownloader fix details.
- **Updated [[infrastructure/proxmox]]** — CT270 entry now shows Nextcloud version and ncdownloader status.
- **Updated [[index.md]]** — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
- No secrets written.
## [2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
- **OmniRoute upstream capacity pressure documented:** `ServiceUnavailableError: Structurally heavy chat request capacity is busy` reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
- **Agent Zero sluggishness confirmed as orchestration overhead:** GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
- **Hermes model verification quirk documented:** Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config `http://omni:20128/v1` is correct — `/v1` is required.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-16) section, bumped `updated`/`verified_on` to 2026-08-16.
- **Updated [[current-state]]** — added Recent Changes (2026-08-16) section, bumped `updated` to 2026-08-16.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-16.
- No new pages created. No secrets written.
## [2026-08-18] incident | Mnemosyne capture pipeline broken (active)
- **New page:** `incidents/2026-08-18-mnemosyne-capture-pipeline` — working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
- **Updated [[systems/mnemosyne]]** — added 2026-08-18 incident to History and Evidence, bumped `updated`/`verified_on` to 2026-08-18, added cross-reference to Related.
- **Updated [[current-state]]** — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped `updated` to 2026-08-18.
- **Updated [[index.md]]** — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
- **Marked [[incidents/2026-08-01-mnemosyne-capture-failure]]** — status changed from `active` to `resolved` (that earlier incident was fixed).
- No secrets written.
## [2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
- **Cron job failures documented (Weekly Review session):** Three distinct failure patterns identified in past 7 days:
- `wiki-stocktake-maintenance` (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from `cx/gpt-5.6-luna` → `go-cheap`/`opencode-go/gpt-5.6-luna`, job unpinned. Requires explicit provider/model pinning.
- CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
- Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
- **Updated [[systems/scheduled-tasks]]** — added Known Issues section with cron failure patterns, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[systems/mnemosyne]]** — added Mnemosyne Watchdog Aug 19 incident to History section, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[current-state]]** — added Recent Changes (2026-08-23) section with cron failures summary.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-23.
- No secrets written.
## [2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
- **New page:** `systems/ignis.md` — documented Ignis browser-native Obsidian (Docker `ignis-subpath:0.8.9`, port 8080, Tailscale Serve at `hermes.kangaroo-eel.ts.net/ignis`). Volume mount: `/home/hermes/wiki/agent-estate-wiki:/vaults/wiki`. Verified live: container running, Tailscale route active.
- **Updated [[systems/model-providers]]** — added `command-code` provider (`https://api.commandcode.ai/provider/v1`) to Current Provider Stack. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-19) section documenting Command Code provider, `deepseek-flash-mix` combo, and Tavili search proxy status. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[infrastructure/docker-services]]** — added Ignis compose file and service definition.
- **Updated [[infrastructure/hosts]]** — added Ignis to CT460 row.
- **Updated [[systems/obsidian]]** — added Ignis as browser interface, corrected vault path to `agent-estate-wiki/`, added Quartz static site reference.
- **Updated [[current-state]]** — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
- **Updated [[index.md]]** — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
- No secrets written.
## [2026-08-26] update | Nightly wiki audit — Cron job reconciliation
- **Updated [[systems/scheduled-tasks]]** — reconciled live `hermes cron list` (27 jobs) against wiki table (29 entries). Removed: `Workday start reminder` (no longer in live list), `CT 221 Miner Watchdog` (job `d5516d0a5147` removed; CT221 status unknown), `Review Agent Estate Phase One` (one-time job completed 2026-08-08). Added: `weekly-save-restricted-bot-restart` (`0 3 * * 0`, local, no-agent). Updated count from 29 to 27.
- **Updated [[current-state]]** — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
- **Updated [[index.md]]** — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
- **Known Issues cleanup** — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
- No secrets written.
## [2026-09-02] incident | SFTPGo outage on CT299
- **New page:** `incidents/2026-09-02-sftpgo-outage.md` — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), `tailscaled` died; PVE `pct start` failed due to 5 stale `lxc-attach` processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, `pct stop 299 --skiplock` + `pct start 299`. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`.
- **Updated [[current-state]]** — added Recent Changes (2026-09-02) section, bumped `updated`/`verified_on` to 2026-09-04.
- **Updated [[infrastructure/proxmox]]** — CT299 entry already documented incident; no change needed.
- No secrets written.
## [2026-08-28] update | Department of Communities role and social-media audit
- **New page:** `reports/communities-wa-social-media-audit-2026` — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
- **Updated [[systems/current-profile]]** — recorded current employment context and linked the report.
- **Updated [[index.md]]** — added the report to the Reports section and refreshed the last-updated date.
- No secrets written.
## [2026-09-20] update | Nightly wiki audit — No changes in past 24h
- **No new pages created** — no new systems, decisions, or incidents in the past 24 hours.
- **No existing pages updated** — all wiki content remains current.
- **Freshness audit run:** `freshness-20260920.txt` generated at 03:01 AWST (scheduled cron).
- **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`).
- **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`.
- No secrets written.
## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
- **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`.
- **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`.
- **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30.
- **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
- **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps.
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
- No secrets written.
## [2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
- **No new pages created** — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
- **Updated `systems/scheduled-tasks.md`** — table and counts re-verified against live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, ×5), `1a8e90ea2271` (HTTP 410, model retired upstream, ×7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: `1b23c042e17d` and `c8d52904e1d6` delivery targets now `telegram:-1004321904721:102`; `c0a9fa314542` flagged for a revoked Telegram bot token; bare-`telegram` targets noted on `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`. Known Issues restructured into Resolved / Open / Historical.
- **Updated `systems/messaging-integrations.md`** — new **Delivery target semantics** section: a bare `deliver: "telegram"` resolves to the home DM (`1793951355`), *not* to the primary group, so a group topic ID attached to it hard-fails with `Message thread not found` and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
- **Updated `systems/model-providers.md`** — provider stack rebuilt from live `config.yaml`: 5 custom providers (`omni-101` 5,384 models, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46); `custom:omni` recorded as removed. Default model corrected `resilient-free` → `smart-route`. New **Retired / Unavailable Models** table (retired `qwen3.5:397b`, plan-gated `gpt-5.6-terra`, catalog-missing `deepseek-v4-flash`) with the lesson that a raw model pin never self-recovers.
- **Updated `systems/mnemosyne.md`** — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as `[USER] [Anthony Martin]` memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
- **Updated `systems/hermes-agent.md`** — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
- **Updated `index.md` and `current-state.md`** — last-updated bumped to 2026-10-01 with a Recent Changes block.
- **Vault sync:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under `wiki/`, not at the tree root, and `index.md` sits at the vault root.
- No secrets written.
## [2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split
- **New page:** `incidents/2026-10-01-cliproxy-connection-auth-expired.md` — job `06a0a1c8704f` (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with `HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Documented as its own incident because it is a **credential** fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (`1b23c042e17d`, `79f9bb41ecf5`).
- **Updated `systems/scheduled-tasks.md`** — counts re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **3 → 6**. New on 2026-10-01: `06a0a1c8704f` (HTTP 401), `0e753e9720fd` (HTTP 400 — same NVIDIA model pin as the already-failing `fad5a86979d4`, confirming the shared-pin diagnosis), `066bb5191280` (delivery failed; report itself generated). Streaks advanced: `dca8482e4f76` ×5→×6, `1a8e90ea2271` ×7→×8, `fad5a86979d4` ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
- **Updated `systems/messaging-integrations.md`** — corrected a wrong page fact. The page previously stated that a bad `thread_id` always loses the message. The 2026-10-01 records show **two variants**: `Message thread not found` (hard failure, message lost — the 2026-09-30 case) and `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` (a thread-less fallback send *does* occur, yet the run is still recorded `delivery_outcome: failed`). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-`telegram` jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
- **Updated `systems/current-profile.md`** — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no `~/.hermes/profiles/default/` directory (the default profile is the top-level config; `ops` is the only named sub-profile), the model default is `smart-route` (not `freellmapi-auto`), the default provider is `custom` → `http://100.78.63.20:20128` / OmniRoute (litellm is now only the last fallback entry), and the config is `_config_version: 46`. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
- **Updated `current-state.md`** — new **Recent Changes (2026-10-02)** block; frontmatter dates bumped.
- **Updated `index.md`** — last-updated to 2026-10-02; cron line now notes the failing count.
- **Structural finding — the wiki exists as two trees and they have drifted both ways.** `/home/hermes/workspace/agent-estate-wiki` is canonical: it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build and is what `wiki-maintain.sh` commits. `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault copy the iPad reads (pages under `wiki/`, no git remote). The audit prompt names only the vault path. The trees have also lost pages in *opposite* directions — `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist only in the vault, while `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` and `reports/communities-wa-social-media-audit-2026.md` exist only in canonical — so canonical `index.md` currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to **both** trees.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), failing set still **6** but the composition moved. `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` that the cliprox incident predicted — making it **2 of the 3** `cliprox/terra` jobs confirmed broken, which retires any "one-job fluke" reading. `79f9bb41ecf5` is the last untested job on that pin (next due 2026-10-04 18:00). **Streak recount** against `executions.db`: `dca8482e4f76` ×6→**×7**, but `1a8e90ea2271` ×8→**×7** — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). **Resolved:** `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). **Downgraded:** the GitHub Trending Radar revoked-token claim is now "unverified at last run" — `c0a9fa314542`'s 2026-09-30 19:22 run completed *and delivered*. New Known Issue recording that **a delivered notice is not a successful job** (both cliprox failures read `delivery_outcome: delivered` on a `failed` status).
- **New page `incidents/2026-08-26-work-outlook-basic-auth-disabled.md`** — the `work` Outlook mailbox has been unreachable since **2026-08-26**: himalaya is configured for password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. TCP/TLS to `outlook.office365.com:993` is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
- **Updated `systems/messaging-integrations.md`** — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, `work` unreachable; frontmatter `updated`/`verified_on` bumped. Added the tooling consequence: `email/mail-lookup` resolves the account before searching, so work-mail requests hit this wall and must be reported as *account unreachable*, not "no results found".
- **Updated `systems/hermes-agent.md`** — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to **six jobs across three model faults plus one credential fault**, and a second entry added for the stock-take job being down; the `Email: connected` bullet corrected.
- **Updated `current-state.md`** and **`index.md`** — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
**New pages**
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
**Updated**
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
## [2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json` + `hermes cron list`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **6 → 7**. The one new failure event in the window is `79f9bb41ecf5` (**Weekly Review**, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. It was the last untested job on the `cliprox/terra` pin, so that pin is now **3 of 3 confirmed broken** — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`) and the Known Issues Open list all updated; frontmatter dates bumped.
- **Updated `incidents/2026-10-01-cliproxy-connection-auth-expired.md`** — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still **Open** — needs the OmniRoute connection re-authenticated before any repin.
- **Updated `current-state.md`** — new **Recent Changes (2026-10-05)** block: the cliprox 3/3 settlement, `1a8e90ea2271`'s repeat 410 on 2026-10-04, confirmation that **no interactive sessions occurred in the window** (all 10 active sessions were cron), and a streak-counting caveat.
- **Updated `index.md`** — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
- **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`.
- No secrets written; all credential-shaped values remain referenced by connection ID only.
## [2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0
Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.
**New**
- `incidents/2026-10-05-dead-model-pin-repin.md` — the seven-job dead-model-pin repin. Records the four dead pins (`cliprox/terra` HTTP 401 ×3, `ollama-cloud/qwen3.5:397b` HTTP 410 ×1, `nvidia/deepseek-ai/deepseek-v4-flash` HTTP 400 ×2, `command-code/gpt-5.6-terra` HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (`free-stack` ×6, `smart-route` ×1 on `custom:omni-101`) over another raw model name.
**Updated**
- `systems/scheduled-tasks.md` — Count block rewritten: **0 failing** (was 7), with all 30 `failure_streak` counters at 0; four jobs carry a stale pre-repin `last_status: error` that clears on their next scheduled run. Seven job-table rows updated (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`, `dca8482e4f76`), plus `c0a9fa314542`. New `### Resolved 2026-10-05` subsection in Known Issues; five resolved bullets removed from `### Open`, leaving bare-`telegram` targets, the two-wiki-tree drift, the work-mailbox block, the stale `custom:omni` key and the delivered-notice rule as the standing open set.
- `index.md` — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.
- `current-state.md` — new Recent Changes (2026-10-06) block; frontmatter dates bumped.
**Findings worth the ink**
- **`dca8482e4f76` (Daily Marketing Job Radar) delivered to topic 1030 at 23:56** — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for `79f9bb41ecf5` (23:39) and `06a0a1c8704f` (23:45).
- **GitHub Trending Radar's revoked-token diagnosis is dead.** `c0a9fa314542` ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No `getMe` → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright.
- **`delivery_outcome: delivered` still does not mean the job worked.** Every pre-repin failure recorded `delivered`, because the failure notice is itself the deliverable. Judge on the execution `status` column.
- **A stale notice opened the review.** Anthony was prompted by a `85edf6772bca` notice that had sat in topic `346` since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` and `dca8482e4f76` at 23:33/23:48, superseded by their successful re-runs) and 1 `running` (this audit).
**Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
## [2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.
**Updated**
- `systems/scheduled-tasks.md` — Count block corrected: the stale pre-repin `last_status: error` set is now **three** (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`), not four — `1a8e90ea2271` cleared its own when it ran at 03:00. Added a **Re-verified 2026-10-07 AWST** bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 `completed` / 0 `failed` / 1 `running`; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
- `index.md` — last-updated → 2026-10-07; cron-fleet line re-dated (**0 failing as of 2026-10-07**, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
- `current-state.md` — new Recent Changes (2026-10-07) block; frontmatter dates bumped.
**Findings worth the ink**
- **`1a8e90ea2271` ran itself at 03:00 and proved the repin.** `wiki-stocktake-maintenance` completed **2026-10-06 03:00:31 → 03:03:32** on `custom:omni-101` / `free-stack` — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired `qwen3.5:397b`). It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt` and committed to Gitea as `33f4e95`. Its four HTTP-410 incidents (`6df957a05ec6`, `d284168d3663`, `cda67de5b3e3`, `e31584909bd2`) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. **The daily manual `wiki-maintain.sh` run can stop.**
- **Vault mirror drift found and partly repaired.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
- **Window tally** (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00): 176 `completed`, 0 `failed`, 1 `running` (this audit) — 177 rows. 14 sessions, all `cron`; no interactive (non-cron) sessions, and no new incident opened.
**Vault mirror:** `systems/scheduled-tasks.md`, `systems/skills-index.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
## [2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings
Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.
**Updated**
- `systems/scheduled-tasks.md` — new **Re-verified 2026-10-08 AWST** bullet (30 jobs, 27 enabled / 3 disabled, **0 failing**; 176 `completed` / 1 `failed` / 1 `running`; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New **Resolved 2026-10-07** Known Issues section, plus two new **Open** items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.
- `index.md` — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.
- `current-state.md` — new Recent Changes (2026-10-08) block; frontmatter dates bumped.
**Findings worth the ink**
- **The incident machinery had its first real workout since 2026-09-30 — and it worked.** `62619aa6bcb3` (DHL Tracking) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) was opened 08:00:52, alerted 08:00:58 and **auto-resolved 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed.
- **The tracked parcel has arrived.** The 14:00 run reported waybill `8052838382` as `Delivered`, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed.
- **Stocktake ran automatically for the second consecutive day** (`1a8e90ea2271`, 2026-10-07 03:00, commit `8c63ef2`; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once.
- **Two real defects surfaced from the stocktake's own log.** (1) Step 3 of `wiki-maintain.sh` — the Quartz rebuild — **is skipped on every run**: there is no `quartz.config.ts` and no `npx`, so `public/` is never refreshed and the published site silently ages. (2) `skills-index` **under-reports by one**: 347 `SKILL.md` files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` (the other duplicated basename, `hermes-api-vault`, survives because its two copies sit in different categories). Both are now Open items in [[systems/scheduled-tasks]].
- **Unchanged, still carrying a stale `last_status: error`:** `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-`telegram` targets and the `custom:omni` stale-key references are also unchanged.
**Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
**Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and
produced no entry, so this run covers two days.
**Pages written**
- `incidents/2026-10-09-cron-provider-timeout-storm.md` — new.
- `incidents/2026-10-09-memory-recall-relevance.md` — new.
- `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
- `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
- `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped.
- `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
- `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed.
**Findings worth the ink**
- **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it.
- **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing.
- **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`.
- **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
- **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key).
**Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
## 2026-10-10 — Stocktake (manual run, `wiki-maintain.sh`)
- **`systems/skills-index.md` regenerated** — 346 skills / 70 categories (unchanged count; frontmatter `updated`/`verified_on` refreshed to 2026-10-10). Live on-disk count is 347 `SKILL.md`; the one-skill gap is the known de-dup of `productivity/obsidian-git-sync/obsidian-git-sync/`.
- **Freshness audit** → `raw/stocktake/freshness-20261010.txt`: **62 stale (≥21d), 23 fresh, 7 no-date of 92 pages**. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No `freshness-20261009.txt` exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands.
- **`current-state.md` — new 2026-10-10 section.** The 2026-10-09 timeout storm window re-tallied from `executions.db` (2026-10-09 00:00 → 2026-10-10 00:00 AWST): **163 `completed`, 15 `failed`** — the prior audit recorded 162/15+1 running; the stray `running` row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).
- **DHL monitor re-probed, still enabled.** `62619aa6bcb3` — `enabled: True`, `paused_at: null`, `0 8,11,14,17,20 * * *`, `last_status: ok`. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. **No cron mutation performed by this audit** — retirement stays open item #1.
- **`infrastructure/tailscale.md` verified against `tailscale serve status`** — all 13 tailnet Serve routes plus the `:8443/webhook` Funnel match the page. No drift; not edited.
- **Not verified / left alone:** the Quartz rebuild step (no `quartz.config.ts`, `npx` cannot fetch `quartz` — `public/` was NOT refreshed, confirming the carried open item); `docker ps` (permission denied on `/var/run/docker.sock` from this session, so `infrastructure/docker-services.md` could not be probed and is left stale); the 80d `2026-07-22` cluster (no live source to check against).
- **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices).
No secrets written; job/incident IDs only.
## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`,
10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5.
Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor
`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09
"deleted" claims were false).
**Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**:
`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env
`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and
verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`;
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
**New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit
`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the
`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault.
**Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
**Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`,
`systems/mnemosyne.md`, `systems/messaging-integrations.md`,
`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-11] stocktake | Daily maintenance + live drift corrections
- Ran `wiki-maintain.sh`: skills-index regenerated (346 skills / 70 categories), freshness audit written to `raw/stocktake/freshness-20261011.txt` (62 stale / 24 fresh / 7 no-date of 93 pages)
- **OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed.** Verified via `pct list`, `pct config`, `ss` and `tailscale status`: CT600 `omniroute` (100.88.81.19) is RUNNING and serving `:20128`; CT601 `omni` (100.93.204.4) is STOPPED with `onboot=0`. Hermes resolves `custom:omni` → `omniroute:20128` → CT600, not CT601. systems/omniroute.md rewritten with the live node table.
- **Third OmniRoute node found:** `omniroute-1` at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider `omni-101` (5384 models, default `smart-route`, carries `skills_hub`). Not in `pct list` — identity unconfirmed.
- **Ignis duplicate found:** a second `ignis-ignis-1` container runs on CT200 (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`), separate from CT460's `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them.
- **Tailscale route table corrected** against live `tailscale serve status`: `/agent-estate` target is bare `http://127.0.0.1:8769` (not `.../8769/agent-estate-role-dashboard.html`); previously-missing tailnet-only `:9443` → `127.0.0.1:5173` documented. CT201 routes refreshed (8085 targets + `/steel` 5173 + `/steel-api` 3000).
- **Proxmox inventory rebuilt** from live `pct list` + `qm list`: added CT475 `agentzero` and VM110 `omarchy`; corrected 282 `ubuntu-server-xfce` to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed rather than guessed.
- **Nanobot verified** on CT333: gateway `nanobot-gateway` (PID 2259048) listening on 8765/8766; NOT a systemd `nanobot.service`/`gateway.service` (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified.
- **Docker services page corrected:** the old "Docker runtime not available" note was a CT460 socket-permission artifact (`/var/run/docker.sock` is `root:docker srw-rw----`, `hermes` user not in group `docker`, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo).
- **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping.
- Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.
## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`,
10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed /
0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5;
**0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`).
**Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor
EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it
cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and
[[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts
refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.
**New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB;
`-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]]
(`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point).
**Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from
August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
and the skills-index blurb no longer claims no regeneration since 2026-10-07.
**Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching
(`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill.
**Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`,
`queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`,
`current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections
**Script run.** `wiki-maintain.sh` completed all three stages: skills-index regenerated from the live
skill tree (**346 skills** across 70 categories, unchanged count), freshness audit written to
`raw/stocktake/freshness-20261012.txt` (**54 stale / 32 fresh / 7 no-date of 93 pages**), and the Quartz
step took its documented no-`quartz.config.ts` path (build skipped, `public/` unchanged — expected, not a
failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and
written its artifacts, so results were read from the report rather than assumed.
**Drift corrected from live probes (2 pages):**
1. **[[systems/nanobot]]** — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on
CT333 is **back up**, HTTP `200` on `http://100.122.189.81:8766/`. Port `8900` — the endpoint the
disabled OmniRoute `nanobot` provider connection uses — is **still closed**, so `status` stays
`degraded` rather than flipping to active. SSH to CT333 is unreachable from CT460, so
`whatsapp-bridge.service` was not re-verified.
2. **[[systems/home-assistant]]** — carried `confidence: low` plus a page-wide "cannot confirm HA runs"
caveat from 2026-07-22. Live: HTTP `200` on `http://100.68.139.5:8123/`, Tailscale node
`homeassistant` active and offering an exit node, VMID 100 running per `qm list`. Confidence raised
low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).
**Verified and left stale.** [[systems/openrouter]] and [[systems/nextcloud]] were probed shallowly and
nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.
**Pages touched:** `systems/nanobot.md`, `systems/home-assistant.md`, `systems/skills-index.md`,
`raw/stocktake/freshness-20261012.txt`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes
**Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from
`~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former
compaction of this job left part of the review written but the log entry and cross-page updates
unfinished — completed here.
**Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly
Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its
fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in
`/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is
a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache
files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded
the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in
[[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised
the same call that failed**, and green cron status is not delivery proof. (Credential was present; value
neither recorded nor repeated.)
**Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily
Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`,
383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs`
and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at
**07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis:
**workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired.
**Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on
three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and
WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host
(CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described
the old port model; both pages now match the probe.
**Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started
23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to
2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**
(`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook
Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 —
the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.
**Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at
**03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and
`dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot
clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no
retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]]
updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded
there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry
their rows for up to a week after the fault is gone.
**Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic
**1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned;
0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]].
**Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran
clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70
categories — the known 347-vs-346 on-disk off-by-one persists).
**Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag),
`systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five
job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new
`## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite,
history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md`
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
`126 / 1`, partial).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
perfect script run leaves nothing behind when the LLM turn dies.
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md`
(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt`
(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again
(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item.
**Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0
(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and
guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale
node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a
hypervisor. Live guests recorded from a single `pct list` / `qm list` pass.
**Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**,
yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak`
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never
on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward.
**Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live
`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now
noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`,
`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`,
`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too
(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not
live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified
**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that
table needed no change.
**Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service`
now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
fault, as diagnosed last cycle.
**Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282
`ubuntu-server-xfce` is now **running**, not stopped as recorded.
**Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed /
1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23
completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open
incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06`
(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot
Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled /
3 disabled**.
**Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry
correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer
exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter),
`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header
warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt`
(new), `log.md`.
**Not verified, left as stale:** the remaining 46 stale pages were not probed against a live
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]],
[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
---
## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit
(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered.
**Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).**
- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in
`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string
recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the
rotations.
- **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight).
Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine
model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log
sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
- **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18
enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside
the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model
calls completed in 34 hours**, and the fleet has no job that could even report the second node's health.
- **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs
(both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as
a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident
page and in [[systems/scheduled-tasks]].
- **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political
News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth
New Restaurant was missed outright** (next slot 10-22).
- **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0);
`96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**.
Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth
required); same on `omniroute:20128` / `100.88.81.19`.
- **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`,
`dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`,
`cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`).
**Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d`
(Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795).
- **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**;
last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %).
Script-only Mnemosyne jobs were unaffected.
- **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left
`raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next
slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the
scheduled run.
- **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B);
`systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line);
`index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry).
**Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page.
**Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on
one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are
a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
consecutive window with no interactive session, so every finding below is from live system state, not
conversation.
### Summary
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new).
- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
incident. **No change was made on the node**; nothing was restarted or repinned.
- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
sentinel. Read failures per job, not per row.
- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16;
10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
hand-run would race the scheduled one.
- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
token is Anthony's call, and the secret is not written here.
- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
rows are typed `script`), so `failure_type` still cannot be used to triage.
- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
threshold, so no reindex is due.
**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**);
`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section);
`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note);
`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window.
**Bridge housekeeping (verified 2026-10-18):** every page this run touched was byte-identical between the
authoring tree (`/home/hermes/workspace/agent-estate-wiki`) and the vault copy
(`/home/hermes/wiki/agent-estate-wiki`, pages nested under `wiki/`) *before* the edits, so the seven files
were written to the authoring tree first, committed and pushed (`4137dd6`), then copied into the vault —
sha256 matched on all seven. The vault-only pages (`wiki/systems/ignis.md`, `wiki/systems/nextcloud.md`)
and the vault's own `wiki/raw/stocktake/` series were not touched; nothing was deleted and no one-way sync
was run. **Drift found, not fixed:** `incidents/2026-09-02-sftpgo-outage.md` exists only in the authoring
tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.
**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
an audit one.
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
## [2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST
**Window verdict: one failure all day, and it was a failure mode the wiki had never recorded.** Fleet
**28 jobs (25 enabled / 3 disabled)**; `10-18 = 168 completed / 1 failed` against `10-17 = 163 / 5`;
`10-19` so far `5 / 0` plus this audit. **11 sessions in the window, all `cron` — no interactive
sessions**, so no user decisions arrived to document; every finding below is machine-observed.
- **A model stream repeated itself and Hermes delivered the placeholder to Telegram.** `ecc1fc13655a`
(FreshRSS Midday, 12:00) wrote *"Your request was not processed. Send it again if you still want me to
carry it out."*, **delivered it** (`delivery_outcome: delivered`, 12:08:11) and then died at 12:08:25 on
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
degenerate response.` New page → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The open
question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue)
instead of substituting text on the job's normal route?
- **Mattermost: cause found — CT255 is stopped**, superseding the 10-11 entry's "cause not determined".
Reconnect loop unchanged (`attempt 471` at 00:14:07, `connect timed out after 30s`, 5-minute cadence),
peer offline **29d**, and the host answers neither TCP `:8065` nor ICMP. Platform config still says
`enabled: true`, so the loop will keep logging until the container is started or the block retired.
- **New page: [[systems/push-notifications]] — the bridge is broken and has been for as long as the logs
go back.** `status.json`: `connected: false`, `error: …connection refused (nothing listens there on
this machine)`; the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which
`/etc/hosts` resolves to **`127.0.1.1`** where nothing serves 443, while the gateway itself is healthy
on `0.0.0.0:9119` (`/` → 302) and Serve still maps the name to it. **23,469** occurrences of that string
across `agent.log(.1-3)` + `errors.log(.1-2)`. Fix documented, **not applied**.
- **Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.** `working_memory`
had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18
03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in
the page); 1 unresolved conflict. The 24 h threshold in `mnemosyne-health-check.sh:92` will keep
flagging idle weekends.
- **Stocktake landed and closed the two-day hole:** `1a8e90ea2271` exited 0 on 10-18 03:00,
`freshness-20261018.txt` written, [[systems/skills-index]] regenerated (346 skills / 70 categories;
freshness 47 stale / 41 fresh / 7 no-date) and committed (`dd53e5f`).
- **Carried forward, still open:** the GitHub Trending Radar delivery token (`getMe` → 401, verified
10-18) — its next exercise is **Mon 2026-10-19 09:00**; the single-node `custom:omni-101` topology with
no model-call failover; and the vault drift below.
**Pages touched:** `incidents/2026-10-18-degenerate-stream-canned-substitute.md` (**new**);
`systems/push-notifications.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet +
FreshRSS row + totals); `systems/messaging-integrations.md` (frontmatter + new Mattermost section);
`systems/mnemosyne.md` (frontmatter + new tier/health block); `infrastructure/tailscale.md` (frontmatter +
inventory header + three missing peers + `mattermost` age); `infrastructure/proxmox.md` (CT255 age);
`current-state.md`; `index.md`; `log.md`. **Nothing removed or archived** — no page became obsolete and no
one-way sync was run.
**Bridge housekeeping + vault drift (verified 2026-10-19 00:2x).** Of the ten files this run touched, six
were byte-identical between the authoring tree and the vault copy *before* the edits; two more
(`infrastructure/tailscale.md`, `infrastructure/proxmox.md`) differed only because the vault holds
**older** versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and
proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (`wiki/`
nesting) with sha256 verified per file.
**Vault drift measured:** against authoring `HEAD`, the vault holds **74 identical pages, 11 stale** and
**12 missing**. Stale: `infrastructure/network-map.md`, `infrastructure/tailscale.md`,
`infrastructure/proxmox.md`, `infrastructure/docker-services.md`, `systems/skills-index.md`,
`systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`,
`systems/obsidian.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md`. Missing:
`incidents/2026-09-02-sftpgo-outage.md` (flagged by the 10-18 audit and still absent), the seven
`agents/*` pages, `reports/communities-wa-social-media-audit-2026.md`, `CONTRIBUTING.md`, plus this run's
two new pages (copied immediately after). The vault repo's own HEAD is still `9bacd8e mirror(2026-10-16)`
with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no remote**, so it cannot be
pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here.
No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.
## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19)
- **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to
`status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 —
**9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7
(2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1.
- **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being
carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong
(`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36`
is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**),
`08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**,
not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1
note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now
flagged in the page for confirmation rather than silently overwritten.
- **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`),
not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL`
count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like
with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24.
- **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from
`jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet,
non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected),
`systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section),
`index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`.
- **Nothing removed or archived** — no page became obsolete and no one-way sync was run.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no
interactive sessions, so there were no user-facing decisions to record.
- No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
- **Vault drift (measured — `sha256` per file against `/home/hermes/wiki/agent-estate-wiki`):** 76 identical /
15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512
"missing", but those are build artefacts under `public/`, `package*.json`, `serve.py` and `raw/stocktake/`,
which the vault deliberately does not carry.) **All six pages touched by this audit were copied in with
per-file `sha256` verification.** The stale pages left unmirrored —
`infrastructure/docker-services.md`, `infrastructure/network-map.md`, `systems/current-profile.md`,
`systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`, `systems/obsidian.md`,
`systems/skills-index.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md` — are authoring-tree edits
made from *other* agents' sessions; **a full vault mirror pass remains the open call.**
- **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commits
`95fd7d3` → `766882e` → `8da6ddf` → `0248698`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote.
### 2026-10-21 — nightly wiki audit (cron `cbfe434a3b5e`, window 2026-10-20)
- **A clean day — and a failure this ledger cannot see.** 2026-10-20 closed **168 `completed` / 0 `failed`**
(plus 1 `unknown`), the **first zero-failure day since 2026-10-12**, ending a seven-day run. The `omni-101` 503
fault did **not** recur (last failure row 2026-10-19), so
[[incidents/2026-10-16-omniroute-503-resource-pressure]] returns to **`monitoring`**. **Zero incident rows
opened, 4 closed → unclosed 13 → 9** (8 `alerted` / 1 `detected`).
- **New page: [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]].** The gateway died with no exit path at
**2026-10-20 09:09:01** (last heartbeat 08:50:28, `pid 2496376`, `suspected_oom=False`), orphaning the 08:30
**Gmail Action Digest** run (`08eb86f02d37`, claimed 08:31:03) as **`unknown`** — *"owner exited before a
durable terminal state"*. No delivery row for the slot, **no incident row created**, and the job table still
shows the *previous* slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits
since 2026-08-01, 5 of them clustered on 2026-08-14/15).
- **New page: [[incidents/2026-08-08-gitea-config-backup-push-dead]].** `e27da6c34bbb` reads `ok` and its
executions read `completed` nightly while **nothing has reached Gitea since 2026-08-07 11:00:47** — **75
unpushed commits**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'`. Second instance
of the "green row, dead job" pattern after the GitHub Trending Radar token revocation.
- **Corrected the 2026-10-18 canned-substitute page.** Its "second surface" section attributed the
interactive-session substitution to an `HTTP 404` Model-not-found; the retained log evidence is
`Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … api_calls=1/150
tool_turns=0 response_len=0`. The page now carries the verified trigger and flags the unresolved model
disagreement (session pin `minimax/minimax-m3.1-flash:free` vs `model=qwen3.5:397b` in the turn-end line).
**The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered,
and is recorded as outstanding in [[current-state]].**
- **Pages updated:** [[current-state]] (new Recent Changes section), [[systems/scheduled-tasks]] (new count bullet
— non-clean `last_status` 12 → 9, config-backup row annotated, frontmatter), [[systems/current-profile]] (model
default moved to `nous` / `meituan/longcat-2.5-preview:free`, no `fallbacks` key, pins unchanged),
[[infrastructure/backups]] (the scheduled Gitea push has been dead since 2026-08-08),
[[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to `monitoring` + dated update),
[[incidents/2026-10-18-degenerate-stream-canned-substitute]] (trigger corrected), `index.md`, this log.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **20 of 28 pin OmniRoute**; the **8 unpinned jobs
moved upstream to `nous`** with the global default change — the fleet now runs two upstream paths. **2
interactive sessions in the window** (`talaria-ios` tax-return work; the `desktop` turn served the canned
non-answer) — the first non-cron sessions after weeks of cron-only windows.
- **Nothing removed or archived** — no page became obsolete, and no one-way sync was run.
- **No secrets written.** The gateway token pasted in the 2026-10-20 desktop session was **not** transcribed into
any page; both the incident and current-state entries describe the request without the credential. Host names,
IPs, job/incident IDs, ports and probe status codes only.
## [2026-10-21] stocktake | Scheduled wiki maintenance (cron `1a8e90ea2271`, 03:00 AWST)
- **Maintenance script ran clean** — `wiki-maintain.sh` completed all three steps: skills-index regenerated
(**346 → 347 skills**; new `australian-tax-return-review` under Finance), freshness audit written
(`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed to `public/`,
`hermes-wiki-serve.service` restarted.
- **[[infrastructure/tailscale]] — node table rebuilt from live `tailscale status --json`.** The 2026-10-19
table had drifted again: **`octop` (100.92.163.122) and `springroll` (100.100.153.125) were missing entirely**,
`openmausbot` / `openmausbot-vm` were described in prose but absent from the table body, `omarchymac` had gone
offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on **DNS name** rather
than the JSON `HostName` — the latter is `ct260` for the `freshrss` node, and **both OmniRoute nodes share the
hostname `omniroute`**, which silently collides. Now **51 non-funnel nodes** (50 IPv4 peers + `hermes`).
Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98):
CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
- **[[reports/uber-eats]] — two missing monthly reports recovered.** The generator
(`~/.hermes/cron/scripts/generate-uber-eats-report.py`) wrote only to `~/wiki/reports/uber-eats/` (the `~/wiki`
repo), **never to the canonical `agent-estate-wiki` tree**, so the 2026-08 and 2026-09 reports were missing from
the published wiki. **Root cause fixed:** the script now writes to `~/workspace/agent-estate-wiki/reports/uber-eats/`
and mirrors to the legacy tree. It also **now emits YAML frontmatter** — without it the pages landed in the
"no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing
months copied in and verified against `~/.hermes/uber_eats_report.json` (2026-09: $51.29 / 2 orders; 2026-08:
$211.16 / 7 orders). The **2026-08 previous-month column was corrected** — the original run had no comparison
payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change.
- **[[incidents/2026-08-15-skills-index-drift]] and [[incidents/2026-09-02-sftpgo-outage]] re-verified against live
state** and annotated. The skills-drift page named a `regenerate-skills-index.sh` that **no longer exists on disk**
(regeneration now lives inside `wiki-maintain.sh` step 1/3); the SFTPGo page confirmed **CT299 running** and the
health endpoint returning **HTTP 401** (auth required, service up) — no recurrence.
- **Still stale, left alone deliberately:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source
document `/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists on disk**,
so the audit baseline cannot be re-verified against a live source. Flagged, not edited.
- **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`,
`model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00.
- **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.
## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin
- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar**
(`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after
40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503
resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session
`cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls);
incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure
(171 `completed` / 1 `failed`).
- **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST**
(action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached),
stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by
updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still
dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard
respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log).
The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted
`v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected.
- **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated),
`systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded),
`systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin),
`incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`),
`incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**,
`message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a
concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry,
scheduled-tasks summary, date).
- **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean
`last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran
clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed,
unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**;
2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken
before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and
`current-state.md`.
- **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.
## [2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)
- Ran `~/.hermes/scripts/wiki-maintain.sh` after the audit so the new incident page is actually published,
rather than waiting for the 03:00 stocktake.
- **`systems/skills-index.md` regenerated from the live tree:** **348 skills** (was 347) across 74 top-level
directories; section order now follows the script's category-priority list. Auto-generated — no hand edits.
- **Freshness report `raw/stocktake/freshness-20261022.txt`:** **1 stale / 95 fresh / 6 no-date of 102 pages**.
The only stale page is `reports/communities-wa-social-media-audit-2026.md` (55 d).
- **Quartz build + deploy:** 273 HTML files into `public/`; `hermes-wiki-serve.service` restarted 00:18:05 AWST
(the build wipes and recreates `public/`, so the running server always needs the restart).
- **Verified live:** `http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip` → 200, and
all four of its wikilinks resolve (`systems/omniroute`, `systems/scheduled-tasks`, `systems/hermes-agent`,
`incidents/2026-10-16-omniroute-503-resource-pressure`); the new entry appears twice on the served index.
## [2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)
- **`systems/scheduled-tasks.md`** — new `## Count` bullet for the 2026-10-22 window: **172 `completed` / 0 `failed`**
(+ this audit `running`), zero failures for the second time in three days; fleet unchanged **28 jobs — 25 enabled /
3 disabled**; **five incidents closed in-window and none opened**, so unclosed rows fall **9 → 4**; non-clean
`last_status` falls **9 → 5**; **27 sessions** (15 `cron`, 3 `subagent`, 6 `desktop`, 2 `acp`, 1 `oneshot`).
Previous bullet marked superseded; `updated`/`verified_on` bumped to 2026-10-23, `review_after` to 2026-10-30.
- **`current-state.md`** — new `Recent Changes (2026-10-23)` block at the top: the 172 / 0 window, incidents
**9 → 4** unclosed and non-clean `last_status` **9 → 5**, the 10-21 recount (168 / 1), the stocktake
no-commit anomaly, the skills-index root fix, the Quartz deploy, and the session mix; `updated`/`verified_on`
bumped to 2026-10-23, `review_after` to 2026-10-30.
- **Correction recorded:** a DB recount (by both `started_at` and `claimed_at`, AWST) puts **2026-10-21 at 168
`completed` / 1 `failed`**, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it
(floor 2026-10-08); the delta is written into the page rather than silently changed.
- **`incidents/2026-08-15-skills-index-drift.md` reopened** (`status: resolved` → `active`, `reopened_on:
2026-10-23`). Evidence: the 2026-10-22 stocktake (`1a8e90ea2271`) ran **03:00:28 → 04:38:19 (97 m 58 s, vs
16 m 01 s on 10-21)**, finished `completed` with delivery `suppressed`, and **committed nothing** — its
regenerated `systems/skills-index.md` (mtime 03:00, `## .Archive (41)` section, 347 total) was still
uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo's
`wiki-maintain.sh` (`find`-based, 347 with `.Archive`) and the stale `~/.hermes/scripts/wiki-maintain.sh`
(`rglob`-based, 348 without). Live counts verified: filesystem **348**, with symlinks **376**, minus `.archive/`
(41) and `apple/` (4) = **331**, loader catalogue **329 enabled** (34 hub-installed / 52 builtin / 243 local).
- **Root-cause fix applied:** the repo's `wiki-maintain.sh` now prunes `*/.archive/*` from the skill scan (the
loader does not expose archived skills), prints the archived count as an explicit exclusion, and names
`hermes skills list` as the authoritative loadable figure. **`systems/skills-index.md` regenerated: 306 skills
across 69 categories** (was 347 counting the parked archive). Script fix and regenerated page committed together.
- **Quartz rebuild + deploy:** 102 files parsed, **602 files emitted to `public/`**, deployed, and
`hermes-wiki-serve.service` restarted 00:14:22 AWST.
- **Freshness report `raw/stocktake/freshness-20261023.txt`:** **2 stale / 94 fresh / 6 no-date of 102 pages** —
the long-standing `reports/communities-wa-social-media-audit-2026.md` (56 d) and, newly over the threshold,
`systems/model-providers.md` (22 d, updated 2026-10-01). The latter needs a review pass against the live
OmniRoute catalogue before its next `verified_on`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST
- **New page:** `systems/conduit.md` — the Conduit dashboard plugin + iOS app (voice calls, cards,
push). Plugin tree `~/.hermes/plugins/conduit_push`, version `0.64.0` (HEAD `eaa14be`, no tags
upstream), remote `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device
(iPhone, app `2.0.0`); state/DB files and sizes recorded. Where the wiki previously had no page for
the phone surface at all.
- **New incident:** [[incidents/2026-10-22-conduit-memory-start-timeout]] — three
`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call` warnings
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s `MEMORY_TIMEOUT_S` in the
plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree
to a 60 s default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades
instead of aborting; the dashboard restarted 18:54:43 and **no abort line exists after it** (two
calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). **The patch is uncommitted.**
- **`incidents/2026-10-09-cron-provider-timeout-storm.md` → `status: resolved`.** Its last two rows
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on `1b23c042e17d`) closed **2026-10-23 08:04**
when that weekly job's Friday slot ran clean; the job reads `ok` again. **All 12 rows closed** —
this incident no longer contributes an unclosed row anywhere.
- **Cron ledger:** window **2026-10-23 closed 169 `completed` / 0 `failed`** (+ this audit `running`) —
the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. Fleet **28 jobs (25 enabled /
3 disabled)**, unchanged for a sixth audit. **Unclosed incident rows 4 → 2** (`c0a9fa_764558b5ae83`
`alerted` since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and
`0e753e_e7f8194a7d06` `detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4**
(`c0a9fa314542`, `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Pins unchanged (11 `omni-101` /
`smart-route`, 8 unpinned, 6 `omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1
`deepseek-paid-smart` = this audit). **14 sessions:** 13 `cron` + 1 `desktop` (18:26:12, the Conduit
phone-call session that exposed the fault above).
- **`systems/mnemosyne.md`:** new block recording the 10–45 s cold start and the consumer-timeout rule
(a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold
start itself was **not** optimised — only the budget was raised.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-24`
section), `current-state.md` (new Recent Changes block; 10-23 block marked superseded), `index.md`
(date, new `systems/conduit` and incident lines, refresh of the scheduled-tasks / skills-index /
storm blurbs), and **frontmatter added** to `incidents/2026-07-29-mnemosyne-relay.md` and
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md` — both sat outside the SCHEMA
(rule 4: every page has frontmatter).
- **Freshness:** 102 pages scanned at the last stocktake; **3 carry a `verified_on` older than 21 days**
— `infrastructure/hosts.md`, `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`
(all 94 d, unchanged). **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Note (hygiene):** a second, older copy of this wiki sits at `/home/hermes/wiki/agent-estate-wiki`
(101 pages, 3 incidents, no git remote, last commit `c798304` "mirror(2026-10-22 maintenance)").
It is **not** served — `hermes-wiki-serve.service` serves `public/` from *this* repo — so it is a
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
syncing it, since no script owns it.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled
at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs
(25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows**
(`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) —
**nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3
`desktop`.**
- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be
phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` →
`CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session.
`conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and
has a bundled skill (`calling-the-user`); the call settings were permissive
(`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted
instead of discovered** — not a missing tool, and not one weak model (the free model's answer was
repeated after a switch to `smart-route`).
- **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence
(“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**.
`conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch**
(`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne
Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is
**empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a
call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43
restart.
- **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22
commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the
uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the
`conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified
(`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59);
store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62
`windows` was a **narrower grouping**, corrected rather than read as a collapse.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25`
section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked
superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22
line).
- **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`,
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are
the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a
`+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59
AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only.
## [2026-10-25] stocktake | wiki-maintain.sh + drift correction (tailscale, homepage-dashboard, 2 stale pages)
- **Script ran clean.** `wiki-maintain.sh` completed all three stages at 03:02–03:03 AWST:
`systems/skills-index.md` regenerated (302 skills / 69 categories, 45 archived — the count fell
from 306 because four skills moved to `~/.hermes/skills/.archive/`: `courier-tracking-monitor`,
`pve-vm-console-recovery`, `model-provider-comparison-research`, `live-application-verification` —
all four confirmed present in the archive directory), freshness audit written to
`raw/stocktake/freshness-20261025.txt`, Quartz rebuilt (630 files) and deployed to `public/`,
`hermes-wiki-serve.service` restarted. Verified the served pages return **200** afterwards.
- **Freshness audit at run time:** 2 stale / 99 fresh / 4 no-date of 105 pages. The two stale pages
were both re-probed live and corrected (see below); after correction the audit reports
**0 stale**.
- **`infrastructure/tailscale.md` — node table rebuilt from live `tailscale status --json`.** The
2026-10-21 table had drifted again:
- **`springroll-browser` (100.109.221.91) was missing from the table entirely** — added (online).
- **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline**
(online → idle/offline 3d). `openmausbot`'s m93p LXC (CT1264) is still running but no longer
has the `tailscale` binary on PATH, so the node itself has stopped connecting.
- **Every offline age was stale** and was refreshed: `mattermost` 31d → **35d**, `omni` 64d → **68d**,
`elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `cloudreve` 88d → **92d**,
`anthonys-imac` 27d → **31d**, `cachyos` 47d → **51d**, `conduit` 44d → **48d**,
`freshrss-new` 64d → **68d**, `omarchymac` 3d → **7d**.
- `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it offers
exit-node capability but is not currently serving as one; the old "active; offers exit node"
wording overstated the live state and was corrected.
- `m93p` (100.104.51.98) reads **idle/offline 13m ago** in the peer table, but the host is plainly
reachable (both `pct list` probes and the SSH session succeeded during this audit), so it is
recorded as a Tailscale-side liveness artefact, not a hypervisor outage.
- Container identities confirmed the authoritative way this run: `tailscale ip -4` executed **inside**
each candidate LXC on both `pve` (100.96.100.82) and `m93p` (100.104.51.98), because peer names
and LXC names do not always match. CT209 (cloudreve), CT255 (mattermost), CT555 (elementsynapse)
and CT601 (omni) confirmed **stopped**; CT600 (omniroute, live OmniRoute) and CT101 (m93p)
confirmed **running**.
- Funnel peer count re-checked: still **23** `funnel-ingress-node` peers with IPv6-only addresses
(excluded). Total non-funnel nodes 51 → **52** (51 IPv4 peers + `hermes` itself).
- **`infrastructure/tailscale.md` + `systems/homepage-dashboard.md` — the CT201 route tables were wrong.**
Live `pct exec 201 -- tailscale serve status` shows CT201 serving Tailscale on **both `:443` and
`:8443`**, and the old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead"
note was **false**: no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both
come back empty for `traefik`), and `ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`.
Corrected routes: `/` → 8085 and `/mgr` → 8084 on **both** ports; `/steel` → 5173 and
`/steel-api` → 3000 on **`:443` only** (`/steel` on `:8443` returns **502**; on `:443` it returns
**200**). The old table also routed `:8443` straight to `http://localhost:8082`, which has **no
listener** — the live target is 8085 (`asset_proxy.py`), which itself proxies to the dead 8082 and
returns 502. Both pages corrected. The **CT460** route table was checked and matched live exactly
(12 paths, no drift).
- **`reports/communities-wa-social-media-audit-2026.md` (was 58d stale) — re-verified, not drifted.**
Checked against the live Department of Communities pages: the remit wording, the ten portfolios,
the **1 July 2025** Housing → Department of Housing and Works restructure, the five-minister
structure and the WA Seniors Card carve-out all still hold; the Department site was updated
23 October 2026 and its recent announcements (carers, youth grants, forced adoption) match the
content pillars recorded here. Audience metrics were **not** re-counted and are left as the
2026-08-28 point-in-time figures; `review_after: 2026-11-26` stands.
- **`incidents/2026-08-26-work-outlook-basic-auth-disabled.md` (was 22d stale) — re-probed, still open.**
`himalaya envelope list -a work -s 1` reproduces the identical failure (`cannot build IMAP client`
/ `login not supported`). The tenant still advertises only `XOAuth2` and the account still
authenticates by password, so no remediation has landed. Status stays `open`; `updated` and
`verified_on` bumped to the verification date with the probe output recorded on the page.
- **`systems/scheduled-tasks.md` checked and left unchanged.** Live `~/.hermes/cron/jobs.json` reads
28 jobs — 25 enabled / 3 disabled, matching the page. All 28 job IDs are present on the page.
Non-clean `last_status` is still **4** (`0e753e9720fd` error, `fad5a86979d4` delivery_failed,
`066bb5191280` delivery_failed, `c0a9fa314542` error streak 1) and unclosed incident rows are
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
entry. No drift, nothing to change.
- **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only.
## [2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
- **Fleet was quiet but one job was not.** Live `executions.db`: 2026-10-25 settled **172 `completed` / 2 `failed`**;
**both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times** in the day (08:31:14 →
10:26:37 runaway at **1 h 55 m** vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31,
19:38:53). New signature on the 19:18:41 fire: **`Cron admission missing: the owner never admitted this fire`**.
All six fires — failures included — carry `delivery_outcome: delivered`. **Neither failure created a `cron_incidents`
row**; the page's open-incident count is unchanged at 2. → new `incidents/2026-10-25-gmail-digest-runaway-and-refire.md`
- **Root cause of the missing context compression found: the Nous Portal credential is dead.**
`Nous Portal runtime credentials unavailable: Invalid refresh token` — **1** line 2026-10-22, **3** on 10-24,
**20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the
19:27:13 restart. → new `incidents/2026-10-22-nous-portal-refresh-token-invalid.md`
- **⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.**
A 2026-10-24 19:06:10 switch note reads *"model was just switched from meituan/longcat-2.5-preview:free to
`smart-route` via omni-101"*; live config confirms `custom:omni-101` / `smart-route` (`api_mode:
chat_completions`), `config.yaml` last written 2026-10-25 19:13. A **six-provider top-level `fallback_providers`
chain is live again** (headed by the dead `nous` credential), so the 2026-10-21 "no fallback chain" record is
stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → **the fleet is a single upstream path again**.
Corrected on `systems/current-profile.md`, `systems/model-providers.md`, `concepts/model-routing.md`,
`comparisons/model-routers.md`, `systems/omniroute.md`.
- **Auxiliary routing captured.** At 2026-10-25 19:13 **all eight** `auxiliary:` slots (approval, compression,
curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to
**`no-think/freeapi/auto` on `custom:omni-101`** with `free_only: true` and a `freeapi/deepseek-v4-flash` →
`kg/kilo-auto/free` chain. ⚠ Masking, not fixing — `nous` remains first in `fallback_providers`. New section on
`systems/model-providers.md`.
- **`systems/scheduled-tasks.md` — new 2026-10-26 entry added**; the 2026-10-25 entry marked superseded. Fleet
unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean `last_status` **4 → 5**
(`08eb86f02d37` joined as `delivery_queued` with an empty `last_error`); pins unchanged.
- **`systems/conduit.md` re-probed live.** Version **`0.69.2`** in `plugin.yaml` *and* `dashboard/manifest.json`
(both), HEAD **`942ddb8`** (2026-10-24 15:35:42, *"Pulse: list candidates, decide in code, journal them (#342)"*),
**0 behind `origin/main`**, 24 commits on since the `0.64.0` baseline, working tree still carrying the
uncommitted `dashboard/plugin_api.py` patch. **The dashboard gateway restarted 2026-10-25 18:42:33, so the
upstream commits are now live** — the previous "not live" note is superseded. `conduit-calls.json → history`
is still **empty**; `conduit_show` ran once on 10-25 14:37. Status on
`incidents/2026-10-24-conduit-call-request-refused.md` **stays open** (untested, not fixed).
- **`systems/messaging-integrations.md`** — CT255 still `stopped` (Mattermost offline **36 d**, `tx 312 / rx 0`);
268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not
outage duration).
- **`systems/hermes-agent.md`** — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04,
19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.
- **`index.md`** — header date, the two new incident links, and rewritten `conduit`, `messaging-integrations` and
`scheduled-tasks` summaries.
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
---
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
corrected where reality could be verified. A second script run then rewrote the same-day report with
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with.
- **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**;
skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
distinguished by path.
- **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
CT255 `stopped`, peer offline **36 d**.
- **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
CT460** (8 disabled here, `Important-mail monitor` the reverse).
- **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
return **200** — the 502 claim was re-confirmed, not merely carried over.
- **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
reindex was triggered.
- **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located".
Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
opencode/nous/ollama fallbacks all 200.
- **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26.
- **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
infrastructure condition (CT255 `stopped`), not an update regression.
- **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
(`last_status: ok`) has nothing to fire on.
- **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`.
- **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only.
### Vault mirror drift (2026-10-26, measured not eyeballed)
The vault at `/home/hermes/wiki` tracks **125 files** under **two** conventions:
`agent-estate-wiki/<path>` (28 files — the Obsidian-visible tree the recent audit
commits write) and `agent-estate-wiki/wiki/<path>` (93 files — the fuller mirror).
This run forward-synced **every page it touched** into both conventions where the
directory existed — **22 copies, all sha256-verified against the authoring source**
before the vault commit. Two vault copies differed from the authoring tree
(`infrastructure/tailscale.md`, `systems/skills-index.md` under `wiki/`); both were
confirmed to be **older versions** (2026-10-21 vs authoring HEAD 2026-10-25), so
no direct vault edits were at risk.
**Remaining drift (left for an explicit request — a full mirror pass is bigger than an
audit's mandate):**
| Convention | match | stale | missing |
|---|---|---|---|
| `agent-estate-wiki/<path>` (top-level) | 25 | 2 | 80 |
| `agent-estate-wiki/wiki/<path>` | 34 | 58 | 15 |
The two top-level **stale** files are `current-state.md` and
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
not pushed.
## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed.
- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**.
**Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
1 `ios`, 1 `telegram`).
- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
`incidents/2026-10-16-omniroute-503-resource-pressure.md`.
- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand.
(b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted).
→ `runbooks/job-radar.md`
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
No secrets written.
- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget);
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
unthreaded chat.
## [2026-10-28] audit | Nightly wiki audit — window 2026-10-27 00:00 → 2026-10-28 00:2x AWST: the radar's tool-call budget ran out a second time — and this time the run is recorded as failed
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
budget overrun, and it is the story of the night → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]].
- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88).
Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters.
- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed).
- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302.
- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
header and any state changes are recorded in that page.
- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
## [2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted
- **New page:** [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — stage 1 (`dca8482e4f76`) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed `failed`, `failure_streak: 2`, on *"hit the agent iteration cap before the handoff could be written"*. No handoff file existed (`~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`), so **both stage-2 slots (`5fc292ff0f3b`, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) `completed` with `delivery_outcome: suppressed`** — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a *different* ceiling (tool-call budget on 10-27 vs iteration cap on 10-28).
- **Updated:** [[incidents/2026-10-27-job-radar-budget-exhausted]] (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did **not** close, unclosed rows 2 → 4) · [[runbooks/job-radar]] (10-28 slot write-up; "first real cycle" marked as run-and-empty) · [[systems/scheduled-tasks]] (new 2026-10-29 window bullet; prior bullet demoted to *Prior window*; the two in-window job rows refreshed) · [[systems/browser-backend]] (dated correction: Camofox restored, `browser-steel` plugin removed, endpoint dark, `STEEL_BASE_URL` residue in 10 live processes) · [[comparisons/steel-vs-camoufox]] · [[decisions/rollback-steel-to-camofox]] (2026-10-29 update) · [[systems/skills-index]] (stocktake-run note) · `current-state.md` (new 2026-10-28 section) · [[index]].
- **Stocktake folded, not paged:** `1a8e90ea2271` ran 03:00:28 → 03:37:58, closed `failed` (streak 1, delivery `suppressed`) on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` after regenerating `systems/skills-index.md` (302 skills) and `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**); it left both uncommitted and filed incident `1a8e90_f57ab4cafab8` in state **`detected`** with no alert. The audit committed the two artefacts, clearing the dirty tree.
- **Live probes (2026-10-29 00:3x AWST):** `jobs.json` 29 jobs / 26 enabled / 3 disabled, 5 non-clean `last_status`, pins 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`; `executions.db` — 2026-10-28 settled **168 / 2 / 1 running**, 2026-10-27 now **173 / 1**, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP `:9223` and REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved; no Hindsight service on this box (`:8888` closed, no `hindsight` units) so [[systems/memory-backend]] and [[comparisons/memory-backends]] were left unchanged.
- **Removed/archived:** none. **Secrets written:** none.
- **Unresolved:** the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); `1a8e90_f57ab4cafab8` sits `detected` without an alert; both radar incident rows close only on a clean 07:00 weekday slot.
## [2026-10-29] update | Steel Browser fully removed from the estate — Camofox is the sole backend
- **Root cause of the long-standing `cloud_provider: steel` residue:** the 2026-07-30 rollback disabled the `browser-steel` plugin and set `browser.engine: camofox`, but left `browser.cloud_provider: steel` in `~/.hermes/config.yaml`. Every browser tool call then routed through `_instantiate_explicit_cloud_provider()` and failed with *"no registered browser plugin has that name: steel"*. Because `engine` was already correct, the split config was inert-but-wrong and survived ~3 months of audits.
- **Hermes (CT460):** `browser:` block now reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`, `cdp_url: ""`. The `browser-steel` plugin directory was **deleted**, and the `browser-steel` key was removed from both `plugins.disabled` and `plugins.entries`. `STEEL_BASE_URL` was dropped from `~/.hermes/.env`. **Hermes core source contains zero Steel references**, so there is no Steel fallback path in the tree — returning to Steel now means redeploying it.
- **CT201 (runtipi):** the entire Steel deployment was torn down — containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network` Docker network, the images `steel-browser-api:fixed` and both ghcr.io Steel images, and the project dir `/root/steel-browser` (~6 GB reclaimed; disk 70G → 64G used). The Tailscale Serve routes `/steel` (5173) and `/steel-api` (3000) were deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`. Nothing listens on 3000, 5173 or 9223.
- **CT333 (clawtest):** a **second, separate and orphaned** Steel install was found and removed — the `steel-browser` container (`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image. It had no Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on 127.0.0.1:9515), no reference from Nanobot's config or the CLI-anything browser harness; its last CDP navigation was 2026-10-20, it received no requests after 2026-10-21, and its page crashed 2026-10-22.
- **Estate-wide sweep:** every LXC on the PVE host was checked — no Steel containers, images, networks or listeners remain anywhere. Homarr, the homepage dashboard and the runtipi app configs had no Steel references.
- **Gateway restarted 01:21 AWST** to clear `STEEL_BASE_URL` from the inherited process environment; the new gateway tree (PID `3913841`) reads `STEEL_BASE_URL=0`, and Telegram, Discord, ntfy, email and Home Assistant all reconnected cleanly. The dashboard unit and `agentmail-webhook.service` still carry the variable in their inherited environment, but **no core code reads it**.
- **Reversibility preserved:** CT201's compose file and Dockerfile are archived under `/root/_archived-steel/`, plus `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. Backups: `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`, `.env.bak-pre-steel-cleanup-20261028T220947Z`.
- **Camofox verified healthy** after every change: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**, `/tabs` → `{"running":true,"tabs":[]}`, and live navigation confirmed working.
- **Pages corrected (15):** [[systems/browser-backend]] · [[decisions/rollback-steel-to-camofox]] · [[decisions/index]] · [[comparisons/steel-vs-camoufox]] · [[comparisons/camoufox-vs-chrome]] · [[infrastructure/docker-services]] · [[infrastructure/tailscale]] · [[infrastructure/hosts]] · [[systems/homepage-dashboard]] · [[concepts/browser-automation]] · [[runbooks/recover-docker-service]] · [[current-state]] · [[index]] · [[agents/nanobot]] · [[systems/scheduled-tasks]].
- **Corrects the 2026-10-29 00:3x audit entry**, which described Steel as merely "not reachable / endpoint dark". It is **removed**, not down. Per SCHEMA rule 9 the superseded claims are marked, not deleted.
- **Removed/archived:** none. **Secrets written:** none (variable names only; no key or URL values).
- **Unresolved:** the obsolete CT333 line in Nanobot's Obsidian vault (`Areas/Homelab/architecture.md`, "Steel Browser | Clawtest | http://localhost:3000") still stands — that vault is outside this wiki and was not modified. Also noted, unrelated to Steel: `Browser Use viewing unavailable: unsupported Hermes browser resolver` has logged in the gateway since 2026-10-11.
## [2026-10-29] stocktake | General Agent Estate Wiki maintenance — Tailscale table re-verified, and a 23-day-old "never re-authenticated" claim turned out to be false
- **Script:** `wiki-maintain.sh` ran clean (exit 0). `systems/skills-index.md` regenerated from the live skill tree — **302 skills in 69 categories**, with a new note that **45 skills** are parked under `~/.hermes/skills/.archive/` and excluded from the count. (303 `SKILL.md` files exist outside `.archive/`, but `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` is a nested alias duplicate of `productivity/obsidian-git-sync/SKILL.md`, so **302 unique skills** is the correct figure.) Freshness report written to `raw/stocktake/freshness-20261029.txt`; Quartz rebuilt (110 pages → 644 files) and deployed to `public/`.
- **Freshness report:** **2 stale / 105 fresh / 3 no-date** of 110 pages. The 2 stale were `incidents/2026-10-01-cliproxy-connection-auth-expired.md` and `incidents/2026-10-05-dead-model-pin-repin.md`, both at **23d**.
- **Updated [[infrastructure/tailscale]]** — State column rebuilt from `tailscale status --json`; **11 rows changed**. Two were **state transitions**: **`sftpgo` (CT299, pve) is back online** (the 2026-10-28 audit had recorded it `offline <1d`; `tailscale ping` → pong in 0s, `systemctl is-active sftpgo` → `active`, `tailscale ip -4` inside the guest → `100.88.223.70`), and **`m93p` is online despite a stale peer record** (`Online: false` in the JSON, but `tailscale ping` → pong in 1ms and SSH over the tailnet returned `m93p / up 11 days`) — the **second** occurrence of that pattern after 2026-10-26. The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged at **54** (53 live IPv4 peers + `hermes`); JSON reports **76 peers — 53 with a `100.` IPv4, 23 IPv6-only funnel ingress**. **No missing peers, no IP mismatches, CT601 still absent from both hypervisors.** Container identity re-confirmed from **inside** all 34 guests via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; the 3 stopped guests (CT209/CT255/CT555) and the 3 running-but-unconnected guests (CT105/CT1264/CT100) confirmed.
- **Updated [[incidents/2026-10-01-cliproxy-connection-auth-expired]]** — the residual claim *"the connection credential itself was never re-authenticated"* is **falsified**. A live `POST /v1/chat/completions` for `cliprox/terra` against `custom:Omniroute:20128` (CT600) returns **HTTP 200** — the connection has since been re-authenticated, independently of the incident's remediation. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`), so the two nodes carry separate credentials. Claim marked superseded rather than deleted (SCHEMA rule 9); the remediation itself (moving 3 jobs off the pin) was never reversed and remains correct.
- **Updated [[incidents/2026-10-05-dead-model-pin-repin]]** — every claim re-probed live: all 7 jobs are still on `custom:omni-101` with routing aliases (`free-stack` ×6, `smart-route` ×1) ✔; both aliases return **HTTP 200** ✔; `ollama-cloud/qwen3.5:397b` still **HTTP 410** (retired 2026-09-25) ✔; `nvidia/deepseek-v4-flash` still **HTTP 400** (absent from the live catalog) ✔; `command-code/gpt-5.6-terra` now returns **HTTP 401** (credential) rather than the original **403** (plan-gated) — the pin is still dead, only the failure signature changed. Frontmatter `updated`/`verified_on` bumped to 2026-10-29 with `created` left at 2026-10-06 as the data period.
- **Updated [[systems/scheduled-tasks]]** — the Known-Issues entry repeating the "never re-authenticated" claim now carries the correction. The page's fleet figures were re-checked against live `jobs.json` and remain accurate: **29 jobs / 26 enabled / 3 disabled**, 5 non-clean `last_status`.
- **Verified, no change needed:** [[systems/homepage-dashboard]] (CT201 `ss -ltn` shows 8084 and 8085 listening, **no 8082** listener, Homarr on 8102 — all as the page claims); CT201 Serve routes (`/` → 8085, `/mgr` → 8084 on **both** `:443` and `:8443`, exactly as documented, Steel routes still gone); CT460 Serve routes (12 paths on `:443` + `:8443/webhook` Funnel + `:9443`, matching the page).
- **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
- **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values).
- **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died
silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated
`wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key;
(B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401**
with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent
call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq`
→ **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL →
**2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard
restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and
08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free.
- **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference,
and `verified_on: 2026-10-30`.
- **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table +
why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second
*superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`).
- **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
*superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar
rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues.
- **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed →
delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed.
- **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and
**[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12
(3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`,
`dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed`
at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed.
- **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors.
- **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since
2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**;
non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak`
reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged.
- **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap.
- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
key or token value.
## 2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state
- **Ran `wiki-maintain.sh`.** Regenerated `systems/skills-index.md` (**302 skills / 69 categories** — unchanged; the
generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote
`raw/stocktake/freshness-20261030.txt` (**0 stale / 109 fresh / 3 no-date of 112 pages**), rebuilt Quartz
(654 files) and deployed to `public/`.
- **Freshness is not accuracy.** The age scan found nothing, but a live-vs-page diff against the running estate found
real drift on five pages. All corrections below were verified against live sources before being written.
- **[[infrastructure/tailscale]] — node table rebuilt.** `tailscale status --json` now reports **50 non-funnel nodes**
(49 live IPv4 peers + `hermes`), not the 54 the page claimed. **2 IP changes** (`anthonys-mac-mini` 100.74.17.15 →
100.76.60.37; `omarchymac` 100.84.180.69 → 100.126.173.115 — the peer formerly listed as `omarchymac-1`) and **4 nodes
dropped out of the tailnet entirely** (`anthonys-imac`, `cachyos`, `omarchymac-1`, `omni`). Every offline age refreshed.
`m93p` reads `Online: false` for the **third** consecutive audit yet is plainly up (`tailscale ping` → pong, SSH → `up 12
days`), so the row records it online. Total peers **72** (49 IPv4 + 23 IPv6-only funnel). CT460's `:443` route set
matched the page exactly; CT201 serves only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`.
- **[[systems/scheduled-tasks]] — fleet ledger corrected.** Live `jobs.json` shows **29 jobs (25 enabled / 4 disabled)**, not
the 28/25/3 on the page: `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused 2026-10-30 01:04:22** after its
00:01:08 slot closed **`unknown`** on a gateway-connection failure. Non-clean `last_status` is **4** (not 3), and the
standing "every `failure_streak` reads 0" claim was false (`ea79a2107e36` = 1). "Failing (delivery only)" is **2**, not 1
(`fad5a86979d4` joined `066bb5191280`). The retired DHL row was dropped from the job table (it lives in Removed Jobs).
Seven job rows carried **stale `error` claims** for jobs that have since cleared to `ok` — each was marked superseded
with a dated correction rather than rewritten (SCHEMA rule 9). The `executions.db` retention floor note was corrected to
the live **2026-10-24 09:22:36**. Window tallies: **2026-10-29 = 177 / 0** and **2026-10-28 = 169 / 2** (both +1 on the
page — the 23:59 Wiki Audit row had still been `running` at audit time).
- **New incident page: [[incidents/2026-10-30-mnemosyne-health-monitor-paused]].** The Mnemosyne Health Monitor's own
failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) —
`deliveries.db` holds **zero rows for 2026-10-30**, so the incident row is the only trace. The job was **not re-enabled**
and the gateway was **not restarted** by this run; root cause is **unverified** and recorded as such.
- **[[infrastructure/hosts]] — four corrections.** CT201's LAN address is `192.168.178.77`, not `192.168.178.166`; CT450's Hermes Agent is **v0.22.5** (2026.9.24),
local **`bf0203a794a0`** — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both
(a stale *root-level* `/usr/local/bin/hermes` on CT450 still reports v0.19.0 and is **not** the live agent —
the live one is the `hermes`-user install under `~/.hermes/hermes-agent`); the cron count is now 29/25/4; and
the orphaned `omni`
peer has **expired out of the tailnet entirely** (no peer named `omni`, no peer at 100.93.204.4), so the "offline 69d"
description no longer applied. The **skills count of 302 was left alone** — it is correct.
- **[[systems/homepage-dashboard]] — five corrections, and a second pass to fix my own first pass.**
"All Apps" is **114 → 123 live apps**, not 122 (the API dedupes by `(name, normalised href)` across groups;
the literal `All Apps` group in `services.yaml` holds 122 and is not what the tab bar counts). The grid is a
responsive `repeat(auto-fill, minmax(230px, 1fr))` (served `/dash.css` line 3640), not a fixed 5-column grid.
The tab list was wrong — but **my first correction was wrong too** and had to be redone: the live tab bar is
built from the service *groups* (`renderTabs()` iterates `state.groups`), **not** from the `layout:` block in
`settings.yaml`. That block and its `tab:`/`columns:` keys are **inert** — `grep -c "Overview" /dash.js` = 0 and
`grep -c "columns" /dash.js` = 0. There is **no "Overview" tab**; the live tabs are `☆ Pinned`, `All (123)` and
the **14 groups** (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20,
Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved
theme is **`{"pair":"Boarding Pass","mode":"light"}`** — one of **59** theme pairs; its light-mode amber is
`#795000`, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. **`manager.py` runs inside
the `homepage-manager` container** (its container `Cmd` is the `pip install … && python3 /opt/homepage/manager.py`
line, confirmed via `docker inspect` and `docker top`); only **`asset_proxy.py`** is a host process
(`0.0.0.0:8085 users:(("python3",pid=4038))`). And the "no Traefik container exists" evidence was wrong — the
`runtipi-reverse-proxy` Traefik container exists but is **stopped** (`Exited (255) 2 months ago`); the *conclusion*
that Traefik does not occupy 443 stands. The same evidence error was corrected in [[infrastructure/tailscale]].
- **[[current-state]]** and **[[index]]** updated with the new incident and the corrected fleet numbers.
- **Verified, not assumed:** CT600 **running** and CT601 **absent from both hypervisors** (`pct config 601` → no such
config on `pve` or `m93p`); the CT201 `:443`/`:8443` Serve tables; CT201 listeners 5173/3000/8082 (**none** — Steel's
removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row
(read from `jobs.json`, `executions.db` — whose `cron_incidents` table lives inside it, the standalone
`cron_incidents.db`/`incidents.db` files are empty stubs — and `deliveries.db`).
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
the steel-removal records across the wiki (confirmed still true).
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
recorded wrong.**
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**;
PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22.
The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**.
Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled.
- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
`last_status`, before writing an incident.**
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01).
- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`.
- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.