Files
agent-estate-wiki/log.md
T
Tony0410 e033a08f17 audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
2026-11-13 00:41:15 +08:00

2550 lines
274 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Wiki Log
> Chronological record of wiki actions. Append-only.
## [2026-07-22] init | Fresh Hermes Systems Wiki
- Created technical wiki at `/home/hermes/wiki`
- Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
- Wrote SCHEMA.md with 11 operating rules
- Wrote index.md as curated Start here dashboard
- Reseeded system/infrastructure/runbook stubs for existing homelab
- Git initialized, local-only
- No secrets, passwords, tokens, or private keys written to vault
## [2026-07-22] audit | External audit pass (Claude, at Anthony's request)
- Merged duplicate pages: `systems/omniRoute.md` (empty stub) -> kept `systems/omniroute.md` (populated); repointed 5 links
- Merged duplicate pages: `runbooks/safely-update-hermes.md` -> kept as `runbooks/update-hermes-safely.md` (fuller content); repointed 2 links
- Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
## [2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
- Fixed a broken `review_after: 0001-01-01` in decisions/keep-openrouter-as-fallback.md
- New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
## [2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
## [2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
## [2026-07-25] operations | Tool Search and gateway resilience
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
- Documented how new skill categories are derived from the first folder below `~/.hermes/skills/`
- Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
- Recorded the I/O-controller delegation limitation so an ineffective `IOWeight` is not mistaken for active protection
- Added health checks, exact effective settings, backup location, and rollback procedure
- No secrets, tokens, or private credentials were written to the wiki
## [2026-07-28] dashboard | Jobs Dashboard feature additions
- Fixed bulk status update (payload shape mismatch: `{ids, status}` → `{ids, updates: {status}}`)
- Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion `Notes / JD Summary`
- Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
- Updated Synergy and Dept of Communities to Phone Screen status
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
## [2026-07-27] incident | Job Radar search and dashboard recovery
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require `MISSING_COVER_LETTER_COUNT=0`
- Recorded the production cron model (`gpt-5.6-terra`) and verified first inherited fallback (`opencode-go/mimo-v2.5`)
- Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both `:443` Serve and `:8443/webhook` Funnel
- Added [[runbooks/job-radar]] and updated scheduled-task, Tailscale and index references
- Verified no secrets, API keys or private credentials were written to the wiki
## [2026-07-29] update | Steel Browser deployment — browser backend switch
- **New page:** `systems/browser-backend.md` — rewritten to reflect Steel Browser as primary, Camoufox as fallback
- **New page:** `comparisons/steel-vs-camoufox.md` — comparison of the two backends
- **Updated:** `infrastructure/hosts.md` — added CT201 and CT460 to verified hosts table
- **Updated:** `infrastructure/docker-services.md` — added Steel Browser compose file
- **Updated:** `infrastructure/tailscale.md` — added CT201 Steel serve routes
- **Updated:** `index.md` — browser-backend link description
- **Updated:** `current-state.md` — added 2026-07-29 changes section
- **Config change:** `browser.cdp_url` set to `http://100.96.244.39:9223` in ops profile
## [2026-07-30] update | Open WebUI updated on CT245
- Updated image to `ghcr.io/open-webui/open-webui:main` (new digest e97bf9531916, Jul 27)
- Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
- Restarted container after OOM hang
- Updated `infrastructure/proxmox.md`
## [2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
- Ran `mnemosyne reindex --yes` to rebuild all embeddings — working memory now at 100%
- Updated health check script (`mnemosyne-health-check.sh`) to report actual embedding coverage via new helper script (`check_embedding_coverage.py`)
- Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
- DB grew from 8.7M to 21M with the new embeddings
## [2026-08-01] incident | Mnemosyne capture pipeline repaired
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no `working_memory` row existed.
- Corrected stringified structured YAML (`sync_roles`, `skip_contexts`, `reflect`, `sleep_threshold`, `ignore_patterns`) and enabled profile isolation; `hermes config check` passed.
- Verified the repair with a second CLI marker that persisted as an exact normalized `[USER]` database row.
- Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route `:9119`; added authoritative identity/preference/workflow slots.
- Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
- Added [[incidents/2026-08-01-mnemosyne-capture-failure]] and refreshed [[systems/mnemosyne]] plus [[systems/memory-backend]].
## [2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
- Rewrote [[comparisons/steel-vs-camoufox]] — full Steel experiment story, timeline, lessons learned
- Rewrote [[systems/browser-backend]] — Camoufox primary, Steel disabled
- Updated [[current-state]] — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
- Updated [[concepts/browser-automation]] — added Steel experiment lesson
- Created [[decisions/rollback-steel-to-camofox]] — formal decision record
- Updated [[systems/scheduled-tasks]] — job count 25, Uber Eats upgrade, GH Trending Daily removed
- Updated [[systems/mnemosyne]] — vector embedding coverage gap documented
- Updated [[systems/model-providers]] — DeepSeek V4 Flash 0731 details
- Created [[reports/uber-eats/index]] — landing page for monthly spend reports
- Updated [[decisions/index]] — linked new decision page
- Updated [[index.md]] — browser description, reports section, last-updated date
- Created nightly wiki audit cron job (23:59 AWST)
## [2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
- Ops profile (`~/.hermes/profiles/ops/config.yaml`): enabled `mnemosyne` plugin with `profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
- Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
- Verified end-to-end: memory writes land correctly in both profiles independently
- Updated [[systems/mnemosyne]] — profile isolation section now documents ops config
- Updated [[current-state]] — added ops profile Mnemosyne entry
- Updated [[runbooks/multiplexer-setup]] — fixed "Elephant ops" → "Tusk ops" in architecture diagram
- Fixed [[systems/memory-backend]] — "Shared surface empty" section renamed to reflect intentional isolation decision
## [2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
- Reworked `deepseek-flash-free` and `good-free` on CT600: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
- Created [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
- Updated [[current-state]] — added OmniRoute free-DeepSeek mitigation to Recent Changes
- Updated [[systems/omniroute]] — added Recent Changes (2026-08-04) section
- Updated [[systems/model-providers]] — added mitigation entry under Notable Model Updates, updated `updated` date
## [2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
- **OmniRoute three-layer fix documented:** npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. `is_active=0` does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-07) section with full root cause analysis
- **Updated [[systems/nanobot]]** — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
- **Updated [[current-state]]** — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
- **Rewrote [[systems/scheduled-tasks]]** — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
- **Wiki audit cron error noted:** model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
- **Updated [[index.md]]** — last-updated date 2026-08-04 → 2026-08-07
## [2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
- **Portainer password reset documented:** CT245 admin user confirmed as `anthony` (not `admin`). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to [[current-state]].
- **SFTPGo security review documented:** CT299 findings — root access, unpinned fingerprint, `//` root path, F1 quota mystery. Added to [[current-state]] and [[infrastructure/proxmox]].
## [2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
- **Cron job model repinning:** 13 jobs repinned from dead/stale provider combos. 3 from `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker from `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named jobs (e.g. `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`) → `smart-route`. Daily Marketing Job Radar → `gpt-5.6-terra`.
- **Hermes UI vite config revert:** Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
- **Phase One estate review completed:** Manual cron run (ID `a9359b7db27b`) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, CT333 Nanobot has live `jobs.json` with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
- **Updated [[current-state]]** — added 2026-08-08 changes section
- **Updated [[systems/scheduled-tasks]]** — Phase One review marked completed, model repinning noted in Known Issues
## [2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
- **Cron job failures documented:** Four jobs failed on 2026-08-10 with two error patterns:
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
- **Root cause identified:** Disk at 83% (11GB free on 65GB), `state.db` at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
- **Updated [[current-state]]** — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
- **Updated [[systems/scheduled-tasks]]** — added session storage failures to Known Issues, updated verified_on to 2026-08-10
- **OpenCode WebSocket proxy errors noted:** Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
- **No new pages created** — all changes applied to existing pages
## [2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
- **OmniRoute AutoRoute variants documented:** `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: `auto/best-free` routed to Antigravity. Updated [[systems/omniroute]].
- **OpenViking memory provider evaluated, NOT adopted:** single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains `memory.provider: mnemosyne`. Added to [[comparisons/memory-backends]].
- **FreeLLMAPI router internals documented:** 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default `freellmapi-auto`. Added to [[comparisons/model-routers]].
- **No new pages created** — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.
## [2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
- **OmniRoute migrated from CT600 to CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. All facts verified live (`pct list`, `grep`, Tailscale DNS).
- **Updated [[systems/omniroute]]** — Current Configuration rewritten for `omni`/CT601, added Recent Changes (2026-08-13), bumped `updated`/`verified_on` to 2026-08-13.
- **Updated [[infrastructure/proxmox]]** — CT600 marked stopped/retired, CT601 added running; bumped `updated`.
- **Updated [[current-state]]** — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
- **Updated [[infrastructure/hosts]]** — OmniRoute proxy entry → `omni`; CT600 reference marked verified/retired.
- **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13.
- No new pages created; all changes applied to existing pages. No secrets written.
## [2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
- **Skills index drift incident:** `systems/skills-index.md` claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating `regenerate-skills-index.sh` script, and adding cron job `1a8e90ea2271` (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See [[incidents/2026-08-15-skills-index-drift]].
- **Updated [[systems/skills-index]]** — added auto-regeneration notice, updated counts (66→330), bumped `updated`/`verified_on` to 2026-08-15.
- **Updated [[systems/scheduled-tasks]]** — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
- **Updated [[systems/hermes-agent]]** — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
- **Updated [[index.md]]** — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
- **Created [[incidents/2026-08-15-skills-index-drift]]** — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
- No secrets written.
## [2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
- **New page:** `systems/nextcloud.md` — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
- **Updated [[current-state]]** — added Recent Changes (2026-08-17) section with ncdownloader fix details.
- **Updated [[infrastructure/proxmox]]** — CT270 entry now shows Nextcloud version and ncdownloader status.
- **Updated [[index.md]]** — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
- No secrets written.
## [2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
- **OmniRoute upstream capacity pressure documented:** `ServiceUnavailableError: Structurally heavy chat request capacity is busy` reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
- **Agent Zero sluggishness confirmed as orchestration overhead:** GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
- **Hermes model verification quirk documented:** Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config `http://omni:20128/v1` is correct — `/v1` is required.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-16) section, bumped `updated`/`verified_on` to 2026-08-16.
- **Updated [[current-state]]** — added Recent Changes (2026-08-16) section, bumped `updated` to 2026-08-16.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-16.
- No new pages created. No secrets written.
## [2026-08-18] incident | Mnemosyne capture pipeline broken (active)
- **New page:** `incidents/2026-08-18-mnemosyne-capture-pipeline` — working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
- **Updated [[systems/mnemosyne]]** — added 2026-08-18 incident to History and Evidence, bumped `updated`/`verified_on` to 2026-08-18, added cross-reference to Related.
- **Updated [[current-state]]** — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped `updated` to 2026-08-18.
- **Updated [[index.md]]** — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
- **Marked [[incidents/2026-08-01-mnemosyne-capture-failure]]** — status changed from `active` to `resolved` (that earlier incident was fixed).
- No secrets written.
## [2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
- **Cron job failures documented (Weekly Review session):** Three distinct failure patterns identified in past 7 days:
- `wiki-stocktake-maintenance` (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from `cx/gpt-5.6-luna` → `go-cheap`/`opencode-go/gpt-5.6-luna`, job unpinned. Requires explicit provider/model pinning.
- CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
- Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
- **Updated [[systems/scheduled-tasks]]** — added Known Issues section with cron failure patterns, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[systems/mnemosyne]]** — added Mnemosyne Watchdog Aug 19 incident to History section, bumped `updated`/`verified_on` to 2026-08-23.
- **Updated [[current-state]]** — added Recent Changes (2026-08-23) section with cron failures summary.
- **Updated [[index.md]]** — bumped last-updated date to 2026-08-23.
- No secrets written.
## [2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
- **New page:** `systems/ignis.md` — documented Ignis browser-native Obsidian (Docker `ignis-subpath:0.8.9`, port 8080, Tailscale Serve at `hermes.kangaroo-eel.ts.net/ignis`). Volume mount: `/home/hermes/wiki/agent-estate-wiki:/vaults/wiki`. Verified live: container running, Tailscale route active.
- **Updated [[systems/model-providers]]** — added `command-code` provider (`https://api.commandcode.ai/provider/v1`) to Current Provider Stack. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-19) section documenting Command Code provider, `deepseek-flash-mix` combo, and Tavili search proxy status. Bumped `updated`/`verified_on` to 2026-08-19.
- **Updated [[infrastructure/docker-services]]** — added Ignis compose file and service definition.
- **Updated [[infrastructure/hosts]]** — added Ignis to CT460 row.
- **Updated [[systems/obsidian]]** — added Ignis as browser interface, corrected vault path to `agent-estate-wiki/`, added Quartz static site reference.
- **Updated [[current-state]]** — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
- **Updated [[index.md]]** — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
- No secrets written.
## [2026-08-26] update | Nightly wiki audit — Cron job reconciliation
- **Updated [[systems/scheduled-tasks]]** — reconciled live `hermes cron list` (27 jobs) against wiki table (29 entries). Removed: `Workday start reminder` (no longer in live list), `CT 221 Miner Watchdog` (job `d5516d0a5147` removed; CT221 status unknown), `Review Agent Estate Phase One` (one-time job completed 2026-08-08). Added: `weekly-save-restricted-bot-restart` (`0 3 * * 0`, local, no-agent). Updated count from 29 to 27.
- **Updated [[current-state]]** — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
- **Updated [[index.md]]** — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
- **Known Issues cleanup** — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
- No secrets written.
## [2026-09-02] incident | SFTPGo outage on CT299
- **New page:** `incidents/2026-09-02-sftpgo-outage.md` — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), `tailscaled` died; PVE `pct start` failed due to 5 stale `lxc-attach` processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, `pct stop 299 --skiplock` + `pct start 299`. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`.
- **Updated [[current-state]]** — added Recent Changes (2026-09-02) section, bumped `updated`/`verified_on` to 2026-09-04.
- **Updated [[infrastructure/proxmox]]** — CT299 entry already documented incident; no change needed.
- No secrets written.
## [2026-08-28] update | Department of Communities role and social-media audit
- **New page:** `reports/communities-wa-social-media-audit-2026` — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
- **Updated [[systems/current-profile]]** — recorded current employment context and linked the report.
- **Updated [[index.md]]** — added the report to the Reports section and refreshed the last-updated date.
- No secrets written.
## [2026-09-20] update | Nightly wiki audit — No changes in past 24h
- **No new pages created** — no new systems, decisions, or incidents in the past 24 hours.
- **No existing pages updated** — all wiki content remains current.
- **Freshness audit run:** `freshness-20260920.txt` generated at 03:01 AWST (scheduled cron).
- **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`).
- **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`.
- No secrets written.
## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
- **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`.
- **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`.
- **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30.
- **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
- **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps.
- **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
- **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed.
- No secrets written.
## [2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
- **No new pages created** — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
- **Updated `systems/scheduled-tasks.md`** — table and counts re-verified against live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, ×5), `1a8e90ea2271` (HTTP 410, model retired upstream, ×7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: `1b23c042e17d` and `c8d52904e1d6` delivery targets now `telegram:-1004321904721:102`; `c0a9fa314542` flagged for a revoked Telegram bot token; bare-`telegram` targets noted on `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`. Known Issues restructured into Resolved / Open / Historical.
- **Updated `systems/messaging-integrations.md`** — new **Delivery target semantics** section: a bare `deliver: "telegram"` resolves to the home DM (`1793951355`), *not* to the primary group, so a group topic ID attached to it hard-fails with `Message thread not found` and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
- **Updated `systems/model-providers.md`** — provider stack rebuilt from live `config.yaml`: 5 custom providers (`omni-101` 5,384 models, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46); `custom:omni` recorded as removed. Default model corrected `resilient-free` → `smart-route`. New **Retired / Unavailable Models** table (retired `qwen3.5:397b`, plan-gated `gpt-5.6-terra`, catalog-missing `deepseek-v4-flash`) with the lesson that a raw model pin never self-recovers.
- **Updated `systems/mnemosyne.md`** — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as `[USER] [Anthony Martin]` memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
- **Updated `systems/hermes-agent.md`** — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
- **Updated `index.md` and `current-state.md`** — last-updated bumped to 2026-10-01 with a Recent Changes block.
- **Vault sync:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under `wiki/`, not at the tree root, and `index.md` sits at the vault root.
- No secrets written.
## [2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split
- **New page:** `incidents/2026-10-01-cliproxy-connection-auth-expired.md` — job `06a0a1c8704f` (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with `HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Documented as its own incident because it is a **credential** fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (`1b23c042e17d`, `79f9bb41ecf5`).
- **Updated `systems/scheduled-tasks.md`** — counts re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **3 → 6**. New on 2026-10-01: `06a0a1c8704f` (HTTP 401), `0e753e9720fd` (HTTP 400 — same NVIDIA model pin as the already-failing `fad5a86979d4`, confirming the shared-pin diagnosis), `066bb5191280` (delivery failed; report itself generated). Streaks advanced: `dca8482e4f76` ×5→×6, `1a8e90ea2271` ×7→×8, `fad5a86979d4` ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
- **Updated `systems/messaging-integrations.md`** — corrected a wrong page fact. The page previously stated that a bad `thread_id` always loses the message. The 2026-10-01 records show **two variants**: `Message thread not found` (hard failure, message lost — the 2026-09-30 case) and `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` (a thread-less fallback send *does* occur, yet the run is still recorded `delivery_outcome: failed`). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-`telegram` jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
- **Updated `systems/current-profile.md`** — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no `~/.hermes/profiles/default/` directory (the default profile is the top-level config; `ops` is the only named sub-profile), the model default is `smart-route` (not `freellmapi-auto`), the default provider is `custom` → `http://100.78.63.20:20128` / OmniRoute (litellm is now only the last fallback entry), and the config is `_config_version: 46`. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
- **Updated `current-state.md`** — new **Recent Changes (2026-10-02)** block; frontmatter dates bumped.
- **Updated `index.md`** — last-updated to 2026-10-02; cron line now notes the failing count.
- **Structural finding — the wiki exists as two trees and they have drifted both ways.** `/home/hermes/workspace/agent-estate-wiki` is canonical: it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build and is what `wiki-maintain.sh` commits. `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault copy the iPad reads (pages under `wiki/`, no git remote). The audit prompt names only the vault path. The trees have also lost pages in *opposite* directions — `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist only in the vault, while `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` and `reports/communities-wa-social-media-audit-2026.md` exist only in canonical — so canonical `index.md` currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to **both** trees.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), failing set still **6** but the composition moved. `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` that the cliprox incident predicted — making it **2 of the 3** `cliprox/terra` jobs confirmed broken, which retires any "one-job fluke" reading. `79f9bb41ecf5` is the last untested job on that pin (next due 2026-10-04 18:00). **Streak recount** against `executions.db`: `dca8482e4f76` ×6→**×7**, but `1a8e90ea2271` ×8→**×7** — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). **Resolved:** `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). **Downgraded:** the GitHub Trending Radar revoked-token claim is now "unverified at last run" — `c0a9fa314542`'s 2026-09-30 19:22 run completed *and delivered*. New Known Issue recording that **a delivered notice is not a successful job** (both cliprox failures read `delivery_outcome: delivered` on a `failed` status).
- **New page `incidents/2026-08-26-work-outlook-basic-auth-disabled.md`** — the `work` Outlook mailbox has been unreachable since **2026-08-26**: himalaya is configured for password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. TCP/TLS to `outlook.office365.com:993` is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
- **Updated `systems/messaging-integrations.md`** — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, `work` unreachable; frontmatter `updated`/`verified_on` bumped. Added the tooling consequence: `email/mail-lookup` resolves the account before searching, so work-mail requests hit this wall and must be reported as *account unreachable*, not "no results found".
- **Updated `systems/hermes-agent.md`** — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to **six jobs across three model faults plus one credential fault**, and a second entry added for the stock-take job being down; the `Email: connected` bullet corrected.
- **Updated `current-state.md`** and **`index.md`** — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
- No secrets written; all credential-shaped values in config remain redacted.
## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
**New pages**
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
**Updated**
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha.
- `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented.
- `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added.
- `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation.
- `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live.
- `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced.
- `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.
- `infrastructure/hosts.md` — CT450 row updated to v0.21.5.
- `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks).
- `runbooks/provider-health.md` — stale v0.19.0 version reference corrected.
- `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated.
**Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`.
## [2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json` + `hermes cron list`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **6 → 7**. The one new failure event in the window is `79f9bb41ecf5` (**Weekly Review**, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. It was the last untested job on the `cliprox/terra` pin, so that pin is now **3 of 3 confirmed broken** — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`) and the Known Issues Open list all updated; frontmatter dates bumped.
- **Updated `incidents/2026-10-01-cliproxy-connection-auth-expired.md`** — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still **Open** — needs the OmniRoute connection re-authenticated before any repin.
- **Updated `current-state.md`** — new **Recent Changes (2026-10-05)** block: the cliprox 3/3 settlement, `1a8e90ea2271`'s repeat 410 on 2026-10-04, confirmation that **no interactive sessions occurred in the window** (all 10 active sessions were cron), and a streak-counting caveat.
- **Updated `index.md`** — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed".
- **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor.
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`.
- No secrets written; all credential-shaped values remain referenced by connection ID only.
## [2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0
Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.
**New**
- `incidents/2026-10-05-dead-model-pin-repin.md` — the seven-job dead-model-pin repin. Records the four dead pins (`cliprox/terra` HTTP 401 ×3, `ollama-cloud/qwen3.5:397b` HTTP 410 ×1, `nvidia/deepseek-ai/deepseek-v4-flash` HTTP 400 ×2, `command-code/gpt-5.6-terra` HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (`free-stack` ×6, `smart-route` ×1 on `custom:omni-101`) over another raw model name.
**Updated**
- `systems/scheduled-tasks.md` — Count block rewritten: **0 failing** (was 7), with all 30 `failure_streak` counters at 0; four jobs carry a stale pre-repin `last_status: error` that clears on their next scheduled run. Seven job-table rows updated (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`, `dca8482e4f76`), plus `c0a9fa314542`. New `### Resolved 2026-10-05` subsection in Known Issues; five resolved bullets removed from `### Open`, leaving bare-`telegram` targets, the two-wiki-tree drift, the work-mailbox block, the stale `custom:omni` key and the delivered-notice rule as the standing open set.
- `index.md` — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.
- `current-state.md` — new Recent Changes (2026-10-06) block; frontmatter dates bumped.
**Findings worth the ink**
- **`dca8482e4f76` (Daily Marketing Job Radar) delivered to topic 1030 at 23:56** — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for `79f9bb41ecf5` (23:39) and `06a0a1c8704f` (23:45).
- **GitHub Trending Radar's revoked-token diagnosis is dead.** `c0a9fa314542` ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No `getMe` → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright.
- **`delivery_outcome: delivered` still does not mean the job worked.** Every pre-repin failure recorded `delivered`, because the failure notice is itself the deliverable. Judge on the execution `status` column.
- **A stale notice opened the review.** Anthony was prompted by a `85edf6772bca` notice that had sat in topic `346` since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` and `dca8482e4f76` at 23:33/23:48, superseded by their successful re-runs) and 1 `running` (this audit).
**Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
## [2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.
**Updated**
- `systems/scheduled-tasks.md` — Count block corrected: the stale pre-repin `last_status: error` set is now **three** (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`), not four — `1a8e90ea2271` cleared its own when it ran at 03:00. Added a **Re-verified 2026-10-07 AWST** bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 `completed` / 0 `failed` / 1 `running`; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.
- `index.md` — last-updated → 2026-10-07; cron-fleet line re-dated (**0 failing as of 2026-10-07**, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.
- `current-state.md` — new Recent Changes (2026-10-07) block; frontmatter dates bumped.
**Findings worth the ink**
- **`1a8e90ea2271` ran itself at 03:00 and proved the repin.** `wiki-stocktake-maintenance` completed **2026-10-06 03:00:31 → 03:03:32** on `custom:omni-101` / `free-stack` — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired `qwen3.5:397b`). It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt` and committed to Gitea as `33f4e95`. Its four HTTP-410 incidents (`6df957a05ec6`, `d284168d3663`, `cda67de5b3e3`, `e31584909bd2`) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. **The daily manual `wiki-maintain.sh` run can stop.**
- **Vault mirror drift found and partly repaired.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item.
- **Window tally** (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00): 176 `completed`, 0 `failed`, 1 `running` (this audit) — 177 rows. 14 sessions, all `cron`; no interactive (non-cron) sessions, and no new incident opened.
**Vault mirror:** `systems/scheduled-tasks.md`, `systems/skills-index.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
## [2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings
Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.
**Updated**
- `systems/scheduled-tasks.md` — new **Re-verified 2026-10-08 AWST** bullet (30 jobs, 27 enabled / 3 disabled, **0 failing**; 176 `completed` / 1 `failed` / 1 `running`; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New **Resolved 2026-10-07** Known Issues section, plus two new **Open** items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.
- `index.md` — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.
- `current-state.md` — new Recent Changes (2026-10-08) block; frontmatter dates bumped.
**Findings worth the ink**
- **The incident machinery had its first real workout since 2026-09-30 — and it worked.** `62619aa6bcb3` (DHL Tracking) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) was opened 08:00:52, alerted 08:00:58 and **auto-resolved 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed.
- **The tracked parcel has arrived.** The 14:00 run reported waybill `8052838382` as `Delivered`, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed.
- **Stocktake ran automatically for the second consecutive day** (`1a8e90ea2271`, 2026-10-07 03:00, commit `8c63ef2`; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once.
- **Two real defects surfaced from the stocktake's own log.** (1) Step 3 of `wiki-maintain.sh` — the Quartz rebuild — **is skipped on every run**: there is no `quartz.config.ts` and no `npx`, so `public/` is never refreshed and the published site silently ages. (2) `skills-index` **under-reports by one**: 347 `SKILL.md` files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` (the other duplicated basename, `hermes-api-vault`, survives because its two copies sit in different categories). Both are now Open items in [[systems/scheduled-tasks]].
- **Unchanged, still carrying a stale `last_status: error`:** `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-`telegram` targets and the `custom:omni` stale-key references are also unchanged.
**Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
**Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and
produced no entry, so this run covers two days.
**Pages written**
- `incidents/2026-10-09-cron-provider-timeout-storm.md` — new.
- `incidents/2026-10-09-memory-recall-relevance.md` — new.
- `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.
- `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident.
- `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped.
- `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.
- `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed.
**Findings worth the ink**
- **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it.
- **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing.
- **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`.
- **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit.
- **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key).
**Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
## 2026-10-10 — Stocktake (manual run, `wiki-maintain.sh`)
- **`systems/skills-index.md` regenerated** — 346 skills / 70 categories (unchanged count; frontmatter `updated`/`verified_on` refreshed to 2026-10-10). Live on-disk count is 347 `SKILL.md`; the one-skill gap is the known de-dup of `productivity/obsidian-git-sync/obsidian-git-sync/`.
- **Freshness audit** → `raw/stocktake/freshness-20261010.txt`: **62 stale (≥21d), 23 fresh, 7 no-date of 92 pages**. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No `freshness-20261009.txt` exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands.
- **`current-state.md` — new 2026-10-10 section.** The 2026-10-09 timeout storm window re-tallied from `executions.db` (2026-10-09 00:00 → 2026-10-10 00:00 AWST): **163 `completed`, 15 `failed`** — the prior audit recorded 162/15+1 running; the stray `running` row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).
- **DHL monitor re-probed, still enabled.** `62619aa6bcb3` — `enabled: True`, `paused_at: null`, `0 8,11,14,17,20 * * *`, `last_status: ok`. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. **No cron mutation performed by this audit** — retirement stays open item #1.
- **`infrastructure/tailscale.md` verified against `tailscale serve status`** — all 13 tailnet Serve routes plus the `:8443/webhook` Funnel match the page. No drift; not edited.
- **Not verified / left alone:** the Quartz rebuild step (no `quartz.config.ts`, `npx` cannot fetch `quartz` — `public/` was NOT refreshed, confirming the carried open item); `docker ps` (permission denied on `/var/run/docker.sock` from this session, so `infrastructure/docker-services.md` could not be probed and is left stale); the 80d `2026-07-22` cluster (no live source to check against).
- **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices).
No secrets written; job/incident IDs only.
## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`,
10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5.
Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor
`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09
"deleted" claims were false).
**Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**:
`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env
`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and
verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`;
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
**New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit
`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the
`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault.
**Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
**Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`,
`systems/mnemosyne.md`, `systems/messaging-integrations.md`,
`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-11] stocktake | Daily maintenance + live drift corrections
- Ran `wiki-maintain.sh`: skills-index regenerated (346 skills / 70 categories), freshness audit written to `raw/stocktake/freshness-20261011.txt` (62 stale / 24 fresh / 7 no-date of 93 pages)
- **OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed.** Verified via `pct list`, `pct config`, `ss` and `tailscale status`: CT600 `omniroute` (100.88.81.19) is RUNNING and serving `:20128`; CT601 `omni` (100.93.204.4) is STOPPED with `onboot=0`. Hermes resolves `custom:omni` → `omniroute:20128` → CT600, not CT601. systems/omniroute.md rewritten with the live node table.
- **Third OmniRoute node found:** `omniroute-1` at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider `omni-101` (5384 models, default `smart-route`, carries `skills_hub`). Not in `pct list` — identity unconfirmed.
- **Ignis duplicate found:** a second `ignis-ignis-1` container runs on CT200 (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`), separate from CT460's `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them.
- **Tailscale route table corrected** against live `tailscale serve status`: `/agent-estate` target is bare `http://127.0.0.1:8769` (not `.../8769/agent-estate-role-dashboard.html`); previously-missing tailnet-only `:9443` → `127.0.0.1:5173` documented. CT201 routes refreshed (8085 targets + `/steel` 5173 + `/steel-api` 3000).
- **Proxmox inventory rebuilt** from live `pct list` + `qm list`: added CT475 `agentzero` and VM110 `omarchy`; corrected 282 `ubuntu-server-xfce` to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed rather than guessed.
- **Nanobot verified** on CT333: gateway `nanobot-gateway` (PID 2259048) listening on 8765/8766; NOT a systemd `nanobot.service`/`gateway.service` (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified.
- **Docker services page corrected:** the old "Docker runtime not available" note was a CT460 socket-permission artifact (`/var/run/docker.sock` is `root:docker srw-rw----`, `hermes` user not in group `docker`, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo).
- **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping.
- Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.
## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`,
10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed /
0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5;
**0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`).
**Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor
EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it
cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and
[[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts
refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.
**New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB;
`-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]]
(`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point).
**Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from
August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
and the skills-index blurb no longer claims no regeneration since 2026-10-07.
**Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching
(`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill.
**Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`,
`queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`,
`current-state.md`, `index.md`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections
**Script run.** `wiki-maintain.sh` completed all three stages: skills-index regenerated from the live
skill tree (**346 skills** across 70 categories, unchanged count), freshness audit written to
`raw/stocktake/freshness-20261012.txt` (**54 stale / 32 fresh / 7 no-date of 93 pages**), and the Quartz
step took its documented no-`quartz.config.ts` path (build skipped, `public/` unchanged — expected, not a
failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and
written its artifacts, so results were read from the report rather than assumed.
**Drift corrected from live probes (2 pages):**
1. **[[systems/nanobot]]** — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on
CT333 is **back up**, HTTP `200` on `http://100.122.189.81:8766/`. Port `8900` — the endpoint the
disabled OmniRoute `nanobot` provider connection uses — is **still closed**, so `status` stays
`degraded` rather than flipping to active. SSH to CT333 is unreachable from CT460, so
`whatsapp-bridge.service` was not re-verified.
2. **[[systems/home-assistant]]** — carried `confidence: low` plus a page-wide "cannot confirm HA runs"
caveat from 2026-07-22. Live: HTTP `200` on `http://100.68.139.5:8123/`, Tailscale node
`homeassistant` active and offering an exit node, VMID 100 running per `qm list`. Confidence raised
low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).
**Verified and left stale.** [[systems/openrouter]] and [[systems/nextcloud]] were probed shallowly and
nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.
**Pages touched:** `systems/nanobot.md`, `systems/home-assistant.md`, `systems/skills-index.md`,
`raw/stocktake/freshness-20261012.txt`, `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes
**Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from
`~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former
compaction of this job left part of the review written but the log entry and cross-page updates
unfinished — completed here.
**Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly
Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its
fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in
`/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is
a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache
files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded
the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in
[[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised
the same call that failed**, and green cron status is not delivery proof. (Credential was present; value
neither recorded nor repeated.)
**Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily
Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`,
383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs`
and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at
**07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis:
**workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired.
**Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on
three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and
WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host
(CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described
the old port model; both pages now match the probe.
**Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started
23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to
2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**
(`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook
Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 —
the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.
**Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at
**03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and
`dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot
clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no
retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]]
updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded
there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry
their rows for up to a week after the fault is gone.
**Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic
**1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned;
0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]].
**Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran
clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70
categories — the known 347-vs-346 on-disk off-by-one persists).
**Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag),
`systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five
job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new
`## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite,
history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md`
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
**Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
`126 / 1`, partial).
**Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
perfect script run leaves nothing behind when the LLM turn dies.
**Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01.
**Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
session.
**Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows).
**Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`.
**Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.
## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md`
(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt`
(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again
(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item.
**Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0
(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and
guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale
node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a
hypervisor. Live guests recorded from a single `pct list` / `qm list` pass.
**Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**,
yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak`
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never
on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward.
**Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live
`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now
noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`,
`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`,
`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too
(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not
live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified
**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that
table needed no change.
**Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service`
now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
fault, as diagnosed last cycle.
**Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282
`ubuntu-server-xfce` is now **running**, not stopped as recorded.
**Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed /
1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23
completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open
incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06`
(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot
Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled /
3 disabled**.
**Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry
correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer
exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter),
`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header
warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt`
(new), `log.md`.
**Not verified, left as stale:** the remaining 46 stale pages were not probed against a live
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]],
[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
---
## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit
(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered.
**Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).**
- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in
`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string
recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the
rotations.
- **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight).
Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine
model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log
sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15).
- **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18
enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
`deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`),
sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside
the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model
calls completed in 34 hours**, and the fleet has no job that could even report the second node's health.
- **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs
(both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as
a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident
page and in [[systems/scheduled-tasks]].
- **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political
News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth
New Restaurant was missed outright** (next slot 10-22).
- **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0);
`96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**.
Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth
required); same on `omniroute:20128` / `100.88.81.19`.
- **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`,
`dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`,
`cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`).
**Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d`
(Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795).
- **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**;
last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %).
Script-only Mnemosyne jobs were unaffected.
- **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left
`raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next
slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the
scheduled run.
- **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive
window with no interactive session.
**Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B);
`systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line);
`index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry).
**Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page.
**Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on
one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are
a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
consecutive window with no interactive session, so every finding below is from live system state, not
conversation.
### Summary
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new).
- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
incident. **No change was made on the node**; nothing was restarted or repinned.
- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
sentinel. Read failures per job, not per row.
- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16;
10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
hand-run would race the scheduled one.
- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
token is Anthony's call, and the secret is not written here.
- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
rows are typed `script`), so `failure_type` still cannot be used to triage.
- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
threshold, so no reindex is due.
**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**);
`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section);
`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note);
`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window.
**Bridge housekeeping (verified 2026-10-18):** every page this run touched was byte-identical between the
authoring tree (`/home/hermes/workspace/agent-estate-wiki`) and the vault copy
(`/home/hermes/wiki/agent-estate-wiki`, pages nested under `wiki/`) *before* the edits, so the seven files
were written to the authoring tree first, committed and pushed (`4137dd6`), then copied into the vault —
sha256 matched on all seven. The vault-only pages (`wiki/systems/ignis.md`, `wiki/systems/nextcloud.md`)
and the vault's own `wiki/raw/stocktake/` series were not touched; nothing was deleted and no one-way sync
was run. **Drift found, not fixed:** `incidents/2026-09-02-sftpgo-outage.md` exists only in the authoring
tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.
**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
an audit one.
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
## [2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST
**Window verdict: one failure all day, and it was a failure mode the wiki had never recorded.** Fleet
**28 jobs (25 enabled / 3 disabled)**; `10-18 = 168 completed / 1 failed` against `10-17 = 163 / 5`;
`10-19` so far `5 / 0` plus this audit. **11 sessions in the window, all `cron` — no interactive
sessions**, so no user decisions arrived to document; every finding below is machine-observed.
- **A model stream repeated itself and Hermes delivered the placeholder to Telegram.** `ecc1fc13655a`
(FreshRSS Midday, 12:00) wrote *"Your request was not processed. Send it again if you still want me to
carry it out."*, **delivered it** (`delivery_outcome: delivered`, 12:08:11) and then died at 12:08:25 on
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
degenerate response.` New page → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The open
question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue)
instead of substituting text on the job's normal route?
- **Mattermost: cause found — CT255 is stopped**, superseding the 10-11 entry's "cause not determined".
Reconnect loop unchanged (`attempt 471` at 00:14:07, `connect timed out after 30s`, 5-minute cadence),
peer offline **29d**, and the host answers neither TCP `:8065` nor ICMP. Platform config still says
`enabled: true`, so the loop will keep logging until the container is started or the block retired.
- **New page: [[systems/push-notifications]] — the bridge is broken and has been for as long as the logs
go back.** `status.json`: `connected: false`, `error: …connection refused (nothing listens there on
this machine)`; the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which
`/etc/hosts` resolves to **`127.0.1.1`** where nothing serves 443, while the gateway itself is healthy
on `0.0.0.0:9119` (`/` → 302) and Serve still maps the name to it. **23,469** occurrences of that string
across `agent.log(.1-3)` + `errors.log(.1-2)`. Fix documented, **not applied**.
- **Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.** `working_memory`
had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18
03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in
the page); 1 unresolved conflict. The 24 h threshold in `mnemosyne-health-check.sh:92` will keep
flagging idle weekends.
- **Stocktake landed and closed the two-day hole:** `1a8e90ea2271` exited 0 on 10-18 03:00,
`freshness-20261018.txt` written, [[systems/skills-index]] regenerated (346 skills / 70 categories;
freshness 47 stale / 41 fresh / 7 no-date) and committed (`dd53e5f`).
- **Carried forward, still open:** the GitHub Trending Radar delivery token (`getMe` → 401, verified
10-18) — its next exercise is **Mon 2026-10-19 09:00**; the single-node `custom:omni-101` topology with
no model-call failover; and the vault drift below.
**Pages touched:** `incidents/2026-10-18-degenerate-stream-canned-substitute.md` (**new**);
`systems/push-notifications.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet +
FreshRSS row + totals); `systems/messaging-integrations.md` (frontmatter + new Mattermost section);
`systems/mnemosyne.md` (frontmatter + new tier/health block); `infrastructure/tailscale.md` (frontmatter +
inventory header + three missing peers + `mattermost` age); `infrastructure/proxmox.md` (CT255 age);
`current-state.md`; `index.md`; `log.md`. **Nothing removed or archived** — no page became obsolete and no
one-way sync was run.
**Bridge housekeeping + vault drift (verified 2026-10-19 00:2x).** Of the ten files this run touched, six
were byte-identical between the authoring tree and the vault copy *before* the edits; two more
(`infrastructure/tailscale.md`, `infrastructure/proxmox.md`) differed only because the vault holds
**older** versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and
proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (`wiki/`
nesting) with sha256 verified per file.
**Vault drift measured:** against authoring `HEAD`, the vault holds **74 identical pages, 11 stale** and
**12 missing**. Stale: `infrastructure/network-map.md`, `infrastructure/tailscale.md`,
`infrastructure/proxmox.md`, `infrastructure/docker-services.md`, `systems/skills-index.md`,
`systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`,
`systems/obsidian.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md`. Missing:
`incidents/2026-09-02-sftpgo-outage.md` (flagged by the 10-18 audit and still absent), the seven
`agents/*` pages, `reports/communities-wa-social-media-audit-2026.md`, `CONTRIBUTING.md`, plus this run's
two new pages (copied immediately after). The vault repo's own HEAD is still `9bacd8e mirror(2026-10-16)`
with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no remote**, so it cannot be
pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here.
No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.
## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19)
- **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to
`status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 —
**9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7
(2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1.
- **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being
carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong
(`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36`
is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**),
`08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**,
not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1
note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now
flagged in the page for confirmation rather than silently overwritten.
- **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`),
not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL`
count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like
with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24.
- **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from
`jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet,
non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected),
`systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section),
`index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`.
- **Nothing removed or archived** — no page became obsolete and no one-way sync was run.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no
interactive sessions, so there were no user-facing decisions to record.
- No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
- **Vault drift (measured — `sha256` per file against `/home/hermes/wiki/agent-estate-wiki`):** 76 identical /
15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512
"missing", but those are build artefacts under `public/`, `package*.json`, `serve.py` and `raw/stocktake/`,
which the vault deliberately does not carry.) **All six pages touched by this audit were copied in with
per-file `sha256` verification.** The stale pages left unmirrored —
`infrastructure/docker-services.md`, `infrastructure/network-map.md`, `systems/current-profile.md`,
`systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`, `systems/obsidian.md`,
`systems/skills-index.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md` — are authoring-tree edits
made from *other* agents' sessions; **a full vault mirror pass remains the open call.**
- **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commits
`95fd7d3` → `766882e` → `8da6ddf` → `0248698`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote.
### 2026-10-21 — nightly wiki audit (cron `cbfe434a3b5e`, window 2026-10-20)
- **A clean day — and a failure this ledger cannot see.** 2026-10-20 closed **168 `completed` / 0 `failed`**
(plus 1 `unknown`), the **first zero-failure day since 2026-10-12**, ending a seven-day run. The `omni-101` 503
fault did **not** recur (last failure row 2026-10-19), so
[[incidents/2026-10-16-omniroute-503-resource-pressure]] returns to **`monitoring`**. **Zero incident rows
opened, 4 closed → unclosed 13 → 9** (8 `alerted` / 1 `detected`).
- **New page: [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]].** The gateway died with no exit path at
**2026-10-20 09:09:01** (last heartbeat 08:50:28, `pid 2496376`, `suspected_oom=False`), orphaning the 08:30
**Gmail Action Digest** run (`08eb86f02d37`, claimed 08:31:03) as **`unknown`** — *"owner exited before a
durable terminal state"*. No delivery row for the slot, **no incident row created**, and the job table still
shows the *previous* slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits
since 2026-08-01, 5 of them clustered on 2026-08-14/15).
- **New page: [[incidents/2026-08-08-gitea-config-backup-push-dead]].** `e27da6c34bbb` reads `ok` and its
executions read `completed` nightly while **nothing has reached Gitea since 2026-08-07 11:00:47** — **75
unpushed commits**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'`. Second instance
of the "green row, dead job" pattern after the GitHub Trending Radar token revocation.
- **Corrected the 2026-10-18 canned-substitute page.** Its "second surface" section attributed the
interactive-session substitution to an `HTTP 404` Model-not-found; the retained log evidence is
`Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … api_calls=1/150
tool_turns=0 response_len=0`. The page now carries the verified trigger and flags the unresolved model
disagreement (session pin `minimax/minimax-m3.1-flash:free` vs `model=qwen3.5:397b` in the turn-end line).
**The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered,
and is recorded as outstanding in [[current-state]].**
- **Pages updated:** [[current-state]] (new Recent Changes section), [[systems/scheduled-tasks]] (new count bullet
— non-clean `last_status` 12 → 9, config-backup row annotated, frontmatter), [[systems/current-profile]] (model
default moved to `nous` / `meituan/longcat-2.5-preview:free`, no `fallbacks` key, pins unchanged),
[[infrastructure/backups]] (the scheduled Gitea push has been dead since 2026-08-08),
[[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to `monitoring` + dated update),
[[incidents/2026-10-18-degenerate-stream-canned-substitute]] (trigger corrected), `index.md`, this log.
- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **20 of 28 pin OmniRoute**; the **8 unpinned jobs
moved upstream to `nous`** with the global default change — the fleet now runs two upstream paths. **2
interactive sessions in the window** (`talaria-ios` tax-return work; the `desktop` turn served the canned
non-answer) — the first non-cron sessions after weeks of cron-only windows.
- **Nothing removed or archived** — no page became obsolete, and no one-way sync was run.
- **No secrets written.** The gateway token pasted in the 2026-10-20 desktop session was **not** transcribed into
any page; both the incident and current-state entries describe the request without the credential. Host names,
IPs, job/incident IDs, ports and probe status codes only.
## [2026-10-21] stocktake | Scheduled wiki maintenance (cron `1a8e90ea2271`, 03:00 AWST)
- **Maintenance script ran clean** — `wiki-maintain.sh` completed all three steps: skills-index regenerated
(**346 → 347 skills**; new `australian-tax-return-review` under Finance), freshness audit written
(`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed to `public/`,
`hermes-wiki-serve.service` restarted.
- **[[infrastructure/tailscale]] — node table rebuilt from live `tailscale status --json`.** The 2026-10-19
table had drifted again: **`octop` (100.92.163.122) and `springroll` (100.100.153.125) were missing entirely**,
`openmausbot` / `openmausbot-vm` were described in prose but absent from the table body, `omarchymac` had gone
offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on **DNS name** rather
than the JSON `HostName` — the latter is `ct260` for the `freshrss` node, and **both OmniRoute nodes share the
hostname `omniroute`**, which silently collides. Now **51 non-funnel nodes** (50 IPv4 peers + `hermes`).
Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98):
CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
- **[[reports/uber-eats]] — two missing monthly reports recovered.** The generator
(`~/.hermes/cron/scripts/generate-uber-eats-report.py`) wrote only to `~/wiki/reports/uber-eats/` (the `~/wiki`
repo), **never to the canonical `agent-estate-wiki` tree**, so the 2026-08 and 2026-09 reports were missing from
the published wiki. **Root cause fixed:** the script now writes to `~/workspace/agent-estate-wiki/reports/uber-eats/`
and mirrors to the legacy tree. It also **now emits YAML frontmatter** — without it the pages landed in the
"no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing
months copied in and verified against `~/.hermes/uber_eats_report.json` (2026-09: $51.29 / 2 orders; 2026-08:
$211.16 / 7 orders). The **2026-08 previous-month column was corrected** — the original run had no comparison
payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change.
- **[[incidents/2026-08-15-skills-index-drift]] and [[incidents/2026-09-02-sftpgo-outage]] re-verified against live
state** and annotated. The skills-drift page named a `regenerate-skills-index.sh` that **no longer exists on disk**
(regeneration now lives inside `wiki-maintain.sh` step 1/3); the SFTPGo page confirmed **CT299 running** and the
health endpoint returning **HTTP 401** (auth required, service up) — no recurrence.
- **Still stale, left alone deliberately:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source
document `/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists on disk**,
so the audit baseline cannot be re-verified against a live source. Flagged, not edited.
- **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`,
`model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00.
- **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.
## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin
- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar**
(`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after
40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503
resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session
`cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls);
incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure
(171 `completed` / 1 `failed`).
- **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST**
(action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached),
stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by
updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still
dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard
respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log).
The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted
`v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected.
- **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated),
`systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded),
`systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin),
`incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`),
`incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**,
`message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a
concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry,
scheduled-tasks summary, date).
- **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean
`last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran
clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed,
unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**;
2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken
before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and
`current-state.md`.
- **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.
## [2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)
- Ran `~/.hermes/scripts/wiki-maintain.sh` after the audit so the new incident page is actually published,
rather than waiting for the 03:00 stocktake.
- **`systems/skills-index.md` regenerated from the live tree:** **348 skills** (was 347) across 74 top-level
directories; section order now follows the script's category-priority list. Auto-generated — no hand edits.
- **Freshness report `raw/stocktake/freshness-20261022.txt`:** **1 stale / 95 fresh / 6 no-date of 102 pages**.
The only stale page is `reports/communities-wa-social-media-audit-2026.md` (55 d).
- **Quartz build + deploy:** 273 HTML files into `public/`; `hermes-wiki-serve.service` restarted 00:18:05 AWST
(the build wipes and recreates `public/`, so the running server always needs the restart).
- **Verified live:** `http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip` → 200, and
all four of its wikilinks resolve (`systems/omniroute`, `systems/scheduled-tasks`, `systems/hermes-agent`,
`incidents/2026-10-16-omniroute-503-resource-pressure`); the new entry appears twice on the served index.
## [2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)
- **`systems/scheduled-tasks.md`** — new `## Count` bullet for the 2026-10-22 window: **172 `completed` / 0 `failed`**
(+ this audit `running`), zero failures for the second time in three days; fleet unchanged **28 jobs — 25 enabled /
3 disabled**; **five incidents closed in-window and none opened**, so unclosed rows fall **9 → 4**; non-clean
`last_status` falls **9 → 5**; **27 sessions** (15 `cron`, 3 `subagent`, 6 `desktop`, 2 `acp`, 1 `oneshot`).
Previous bullet marked superseded; `updated`/`verified_on` bumped to 2026-10-23, `review_after` to 2026-10-30.
- **`current-state.md`** — new `Recent Changes (2026-10-23)` block at the top: the 172 / 0 window, incidents
**9 → 4** unclosed and non-clean `last_status` **9 → 5**, the 10-21 recount (168 / 1), the stocktake
no-commit anomaly, the skills-index root fix, the Quartz deploy, and the session mix; `updated`/`verified_on`
bumped to 2026-10-23, `review_after` to 2026-10-30.
- **Correction recorded:** a DB recount (by both `started_at` and `claimed_at`, AWST) puts **2026-10-21 at 168
`completed` / 1 `failed`**, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it
(floor 2026-10-08); the delta is written into the page rather than silently changed.
- **`incidents/2026-08-15-skills-index-drift.md` reopened** (`status: resolved` → `active`, `reopened_on:
2026-10-23`). Evidence: the 2026-10-22 stocktake (`1a8e90ea2271`) ran **03:00:28 → 04:38:19 (97 m 58 s, vs
16 m 01 s on 10-21)**, finished `completed` with delivery `suppressed`, and **committed nothing** — its
regenerated `systems/skills-index.md` (mtime 03:00, `## .Archive (41)` section, 347 total) was still
uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo's
`wiki-maintain.sh` (`find`-based, 347 with `.Archive`) and the stale `~/.hermes/scripts/wiki-maintain.sh`
(`rglob`-based, 348 without). Live counts verified: filesystem **348**, with symlinks **376**, minus `.archive/`
(41) and `apple/` (4) = **331**, loader catalogue **329 enabled** (34 hub-installed / 52 builtin / 243 local).
- **Root-cause fix applied:** the repo's `wiki-maintain.sh` now prunes `*/.archive/*` from the skill scan (the
loader does not expose archived skills), prints the archived count as an explicit exclusion, and names
`hermes skills list` as the authoritative loadable figure. **`systems/skills-index.md` regenerated: 306 skills
across 69 categories** (was 347 counting the parked archive). Script fix and regenerated page committed together.
- **Quartz rebuild + deploy:** 102 files parsed, **602 files emitted to `public/`**, deployed, and
`hermes-wiki-serve.service` restarted 00:14:22 AWST.
- **Freshness report `raw/stocktake/freshness-20261023.txt`:** **2 stale / 94 fresh / 6 no-date of 102 pages** —
the long-standing `reports/communities-wa-social-media-audit-2026.md` (56 d) and, newly over the threshold,
`systems/model-providers.md` (22 d, updated 2026-10-01). The latter needs a review pass against the live
OmniRoute catalogue before its next `verified_on`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST
- **New page:** `systems/conduit.md` — the Conduit dashboard plugin + iOS app (voice calls, cards,
push). Plugin tree `~/.hermes/plugins/conduit_push`, version `0.64.0` (HEAD `eaa14be`, no tags
upstream), remote `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device
(iPhone, app `2.0.0`); state/DB files and sizes recorded. Where the wiki previously had no page for
the phone surface at all.
- **New incident:** [[incidents/2026-10-22-conduit-memory-start-timeout]] — three
`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call` warnings
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s `MEMORY_TIMEOUT_S` in the
plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree
to a 60 s default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades
instead of aborting; the dashboard restarted 18:54:43 and **no abort line exists after it** (two
calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). **The patch is uncommitted.**
- **`incidents/2026-10-09-cron-provider-timeout-storm.md` → `status: resolved`.** Its last two rows
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on `1b23c042e17d`) closed **2026-10-23 08:04**
when that weekly job's Friday slot ran clean; the job reads `ok` again. **All 12 rows closed** —
this incident no longer contributes an unclosed row anywhere.
- **Cron ledger:** window **2026-10-23 closed 169 `completed` / 0 `failed`** (+ this audit `running`) —
the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. Fleet **28 jobs (25 enabled /
3 disabled)**, unchanged for a sixth audit. **Unclosed incident rows 4 → 2** (`c0a9fa_764558b5ae83`
`alerted` since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and
`0e753e_e7f8194a7d06` `detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4**
(`c0a9fa314542`, `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Pins unchanged (11 `omni-101` /
`smart-route`, 8 unpinned, 6 `omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1
`deepseek-paid-smart` = this audit). **14 sessions:** 13 `cron` + 1 `desktop` (18:26:12, the Conduit
phone-call session that exposed the fault above).
- **`systems/mnemosyne.md`:** new block recording the 10–45 s cold start and the consumer-timeout rule
(a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold
start itself was **not** optimised — only the budget was raised.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-24`
section), `current-state.md` (new Recent Changes block; 10-23 block marked superseded), `index.md`
(date, new `systems/conduit` and incident lines, refresh of the scheduled-tasks / skills-index /
storm blurbs), and **frontmatter added** to `incidents/2026-07-29-mnemosyne-relay.md` and
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md` — both sat outside the SCHEMA
(rule 4: every page has frontmatter).
- **Freshness:** 102 pages scanned at the last stocktake; **3 carry a `verified_on` older than 21 days**
— `infrastructure/hosts.md`, `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`
(all 94 d, unchanged). **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Note (hygiene):** a second, older copy of this wiki sits at `/home/hermes/wiki/agent-estate-wiki`
(101 pages, 3 incidents, no git remote, last commit `c798304` "mirror(2026-10-22 maintenance)").
It is **not** served — `hermes-wiki-serve.service` serves `public/` from *this* repo — so it is a
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
syncing it, since no script owns it.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled
at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs
(25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows**
(`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) —
**nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3
`desktop`.**
- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be
phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` →
`CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session.
`conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and
has a bundled skill (`calling-the-user`); the call settings were permissive
(`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted
instead of discovered** — not a missing tool, and not one weak model (the free model's answer was
repeated after a switch to `smart-route`).
- **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence
(“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**.
`conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch**
(`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne
Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is
**empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a
call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43
restart.
- **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22
commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the
uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the
`conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified
(`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59);
store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62
`windows` was a **narrower grouping**, corrected rather than read as a collapse.
- **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25`
section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked
superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22
line).
- **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`,
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are
the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system
obsolete, no decision reversed in the window.
- **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a
`+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59
AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`.
- **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only.
## [2026-10-25] stocktake | wiki-maintain.sh + drift correction (tailscale, homepage-dashboard, 2 stale pages)
- **Script ran clean.** `wiki-maintain.sh` completed all three stages at 03:02–03:03 AWST:
`systems/skills-index.md` regenerated (302 skills / 69 categories, 45 archived — the count fell
from 306 because four skills moved to `~/.hermes/skills/.archive/`: `courier-tracking-monitor`,
`pve-vm-console-recovery`, `model-provider-comparison-research`, `live-application-verification` —
all four confirmed present in the archive directory), freshness audit written to
`raw/stocktake/freshness-20261025.txt`, Quartz rebuilt (630 files) and deployed to `public/`,
`hermes-wiki-serve.service` restarted. Verified the served pages return **200** afterwards.
- **Freshness audit at run time:** 2 stale / 99 fresh / 4 no-date of 105 pages. The two stale pages
were both re-probed live and corrected (see below); after correction the audit reports
**0 stale**.
- **`infrastructure/tailscale.md` — node table rebuilt from live `tailscale status --json`.** The
2026-10-21 table had drifted again:
- **`springroll-browser` (100.109.221.91) was missing from the table entirely** — added (online).
- **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline**
(online → idle/offline 3d). `openmausbot`'s m93p LXC (CT1264) is still running but no longer
has the `tailscale` binary on PATH, so the node itself has stopped connecting.
- **Every offline age was stale** and was refreshed: `mattermost` 31d → **35d**, `omni` 64d → **68d**,
`elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `cloudreve` 88d → **92d**,
`anthonys-imac` 27d → **31d**, `cachyos` 47d → **51d**, `conduit` 44d → **48d**,
`freshrss-new` 64d → **68d**, `omarchymac` 3d → **7d**.
- `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it offers
exit-node capability but is not currently serving as one; the old "active; offers exit node"
wording overstated the live state and was corrected.
- `m93p` (100.104.51.98) reads **idle/offline 13m ago** in the peer table, but the host is plainly
reachable (both `pct list` probes and the SSH session succeeded during this audit), so it is
recorded as a Tailscale-side liveness artefact, not a hypervisor outage.
- Container identities confirmed the authoritative way this run: `tailscale ip -4` executed **inside**
each candidate LXC on both `pve` (100.96.100.82) and `m93p` (100.104.51.98), because peer names
and LXC names do not always match. CT209 (cloudreve), CT255 (mattermost), CT555 (elementsynapse)
and CT601 (omni) confirmed **stopped**; CT600 (omniroute, live OmniRoute) and CT101 (m93p)
confirmed **running**.
- Funnel peer count re-checked: still **23** `funnel-ingress-node` peers with IPv6-only addresses
(excluded). Total non-funnel nodes 51 → **52** (51 IPv4 peers + `hermes` itself).
- **`infrastructure/tailscale.md` + `systems/homepage-dashboard.md` — the CT201 route tables were wrong.**
Live `pct exec 201 -- tailscale serve status` shows CT201 serving Tailscale on **both `:443` and
`:8443`**, and the old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead"
note was **false**: no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both
come back empty for `traefik`), and `ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`.
Corrected routes: `/` → 8085 and `/mgr` → 8084 on **both** ports; `/steel` → 5173 and
`/steel-api` → 3000 on **`:443` only** (`/steel` on `:8443` returns **502**; on `:443` it returns
**200**). The old table also routed `:8443` straight to `http://localhost:8082`, which has **no
listener** — the live target is 8085 (`asset_proxy.py`), which itself proxies to the dead 8082 and
returns 502. Both pages corrected. The **CT460** route table was checked and matched live exactly
(12 paths, no drift).
- **`reports/communities-wa-social-media-audit-2026.md` (was 58d stale) — re-verified, not drifted.**
Checked against the live Department of Communities pages: the remit wording, the ten portfolios,
the **1 July 2025** Housing → Department of Housing and Works restructure, the five-minister
structure and the WA Seniors Card carve-out all still hold; the Department site was updated
23 October 2026 and its recent announcements (carers, youth grants, forced adoption) match the
content pillars recorded here. Audience metrics were **not** re-counted and are left as the
2026-08-28 point-in-time figures; `review_after: 2026-11-26` stands.
- **`incidents/2026-08-26-work-outlook-basic-auth-disabled.md` (was 22d stale) — re-probed, still open.**
`himalaya envelope list -a work -s 1` reproduces the identical failure (`cannot build IMAP client`
/ `login not supported`). The tenant still advertises only `XOAuth2` and the account still
authenticates by password, so no remediation has landed. Status stays `open`; `updated` and
`verified_on` bumped to the verification date with the probe output recorded on the page.
- **`systems/scheduled-tasks.md` checked and left unchanged.** Live `~/.hermes/cron/jobs.json` reads
28 jobs — 25 enabled / 3 disabled, matching the page. All 28 job IDs are present on the page.
Non-clean `last_status` is still **4** (`0e753e9720fd` error, `fad5a86979d4` delivery_failed,
`066bb5191280` delivery_failed, `c0a9fa314542` error streak 1) and unclosed incident rows are
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
entry. No drift, nothing to change.
- **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only.
## [2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
- **Fleet was quiet but one job was not.** Live `executions.db`: 2026-10-25 settled **172 `completed` / 2 `failed`**;
**both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times** in the day (08:31:14 →
10:26:37 runaway at **1 h 55 m** vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31,
19:38:53). New signature on the 19:18:41 fire: **`Cron admission missing: the owner never admitted this fire`**.
All six fires — failures included — carry `delivery_outcome: delivered`. **Neither failure created a `cron_incidents`
row**; the page's open-incident count is unchanged at 2. → new `incidents/2026-10-25-gmail-digest-runaway-and-refire.md`
- **Root cause of the missing context compression found: the Nous Portal credential is dead.**
`Nous Portal runtime credentials unavailable: Invalid refresh token` — **1** line 2026-10-22, **3** on 10-24,
**20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the
19:27:13 restart. → new `incidents/2026-10-22-nous-portal-refresh-token-invalid.md`
- **⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.**
A 2026-10-24 19:06:10 switch note reads *"model was just switched from meituan/longcat-2.5-preview:free to
`smart-route` via omni-101"*; live config confirms `custom:omni-101` / `smart-route` (`api_mode:
chat_completions`), `config.yaml` last written 2026-10-25 19:13. A **six-provider top-level `fallback_providers`
chain is live again** (headed by the dead `nous` credential), so the 2026-10-21 "no fallback chain" record is
stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → **the fleet is a single upstream path again**.
Corrected on `systems/current-profile.md`, `systems/model-providers.md`, `concepts/model-routing.md`,
`comparisons/model-routers.md`, `systems/omniroute.md`.
- **Auxiliary routing captured.** At 2026-10-25 19:13 **all eight** `auxiliary:` slots (approval, compression,
curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to
**`no-think/freeapi/auto` on `custom:omni-101`** with `free_only: true` and a `freeapi/deepseek-v4-flash` →
`kg/kilo-auto/free` chain. ⚠ Masking, not fixing — `nous` remains first in `fallback_providers`. New section on
`systems/model-providers.md`.
- **`systems/scheduled-tasks.md` — new 2026-10-26 entry added**; the 2026-10-25 entry marked superseded. Fleet
unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean `last_status` **4 → 5**
(`08eb86f02d37` joined as `delivery_queued` with an empty `last_error`); pins unchanged.
- **`systems/conduit.md` re-probed live.** Version **`0.69.2`** in `plugin.yaml` *and* `dashboard/manifest.json`
(both), HEAD **`942ddb8`** (2026-10-24 15:35:42, *"Pulse: list candidates, decide in code, journal them (#342)"*),
**0 behind `origin/main`**, 24 commits on since the `0.64.0` baseline, working tree still carrying the
uncommitted `dashboard/plugin_api.py` patch. **The dashboard gateway restarted 2026-10-25 18:42:33, so the
upstream commits are now live** — the previous "not live" note is superseded. `conduit-calls.json → history`
is still **empty**; `conduit_show` ran once on 10-25 14:37. Status on
`incidents/2026-10-24-conduit-call-request-refused.md` **stays open** (untested, not fixed).
- **`systems/messaging-integrations.md`** — CT255 still `stopped` (Mattermost offline **36 d**, `tx 312 / rx 0`);
268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not
outage duration).
- **`systems/hermes-agent.md`** — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04,
19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.
- **`index.md`** — header date, the two new incident links, and rewritten `conduit`, `messaging-integrations` and
`scheduled-tasks` summaries.
- **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
- **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
provider entry (still named by 2 jobs — Anthony's call, not the wiki's).
---
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
corrected where reality could be verified. A second script run then rewrote the same-day report with
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with.
- **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**;
skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
distinguished by path.
- **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
CT255 `stopped`, peer offline **36 d**.
- **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
CT460** (8 disabled here, `Important-mail monitor` the reverse).
- **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
return **200** — the 502 claim was re-confirmed, not merely carried over.
- **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
reindex was triggered.
- **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located".
Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
opencode/nous/ollama fallbacks all 200.
- **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26.
- **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
infrastructure condition (CT255 `stopped`), not an update regression.
- **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
(`last_status: ok`) has nothing to fire on.
- **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`.
- **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only.
### Vault mirror drift (2026-10-26, measured not eyeballed)
The vault at `/home/hermes/wiki` tracks **125 files** under **two** conventions:
`agent-estate-wiki/<path>` (28 files — the Obsidian-visible tree the recent audit
commits write) and `agent-estate-wiki/wiki/<path>` (93 files — the fuller mirror).
This run forward-synced **every page it touched** into both conventions where the
directory existed — **22 copies, all sha256-verified against the authoring source**
before the vault commit. Two vault copies differed from the authoring tree
(`infrastructure/tailscale.md`, `systems/skills-index.md` under `wiki/`); both were
confirmed to be **older versions** (2026-10-21 vs authoring HEAD 2026-10-25), so
no direct vault edits were at risk.
**Remaining drift (left for an explicit request — a full mirror pass is bigger than an
audit's mandate):**
| Convention | match | stale | missing |
|---|---|---|---|
| `agent-estate-wiki/<path>` (top-level) | 25 | 2 | 80 |
| `agent-estate-wiki/wiki/<path>` | 34 | 58 | 15 |
The two top-level **stale** files are `current-state.md` and
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
not pushed.
## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed.
- **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**.
**Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
1 `ios`, 1 `telegram`).
- **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
`incidents/2026-10-16-omniroute-503-resource-pressure.md`.
- **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand.
(b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted).
→ `runbooks/job-radar.md`
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
No secrets written.
- **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget);
whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
unthreaded chat.
## [2026-10-28] audit | Nightly wiki audit — window 2026-10-27 00:00 → 2026-10-28 00:2x AWST: the radar's tool-call budget ran out a second time — and this time the run is recorded as failed
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
budget overrun, and it is the story of the night → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]].
- **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88).
Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters.
- **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed).
- **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302.
- **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
header and any state changes are recorded in that page.
- **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
- **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
## [2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted
- **New page:** [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — stage 1 (`dca8482e4f76`) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed `failed`, `failure_streak: 2`, on *"hit the agent iteration cap before the handoff could be written"*. No handoff file existed (`~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`), so **both stage-2 slots (`5fc292ff0f3b`, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) `completed` with `delivery_outcome: suppressed`** — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a *different* ceiling (tool-call budget on 10-27 vs iteration cap on 10-28).
- **Updated:** [[incidents/2026-10-27-job-radar-budget-exhausted]] (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did **not** close, unclosed rows 2 → 4) · [[runbooks/job-radar]] (10-28 slot write-up; "first real cycle" marked as run-and-empty) · [[systems/scheduled-tasks]] (new 2026-10-29 window bullet; prior bullet demoted to *Prior window*; the two in-window job rows refreshed) · [[systems/browser-backend]] (dated correction: Camofox restored, `browser-steel` plugin removed, endpoint dark, `STEEL_BASE_URL` residue in 10 live processes) · [[comparisons/steel-vs-camoufox]] · [[decisions/rollback-steel-to-camofox]] (2026-10-29 update) · [[systems/skills-index]] (stocktake-run note) · `current-state.md` (new 2026-10-28 section) · [[index]].
- **Stocktake folded, not paged:** `1a8e90ea2271` ran 03:00:28 → 03:37:58, closed `failed` (streak 1, delivery `suppressed`) on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` after regenerating `systems/skills-index.md` (302 skills) and `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**); it left both uncommitted and filed incident `1a8e90_f57ab4cafab8` in state **`detected`** with no alert. The audit committed the two artefacts, clearing the dirty tree.
- **Live probes (2026-10-29 00:3x AWST):** `jobs.json` 29 jobs / 26 enabled / 3 disabled, 5 non-clean `last_status`, pins 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`; `executions.db` — 2026-10-28 settled **168 / 2 / 1 running**, 2026-10-27 now **173 / 1**, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP `:9223` and REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved; no Hindsight service on this box (`:8888` closed, no `hindsight` units) so [[systems/memory-backend]] and [[comparisons/memory-backends]] were left unchanged.
- **Removed/archived:** none. **Secrets written:** none.
- **Unresolved:** the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); `1a8e90_f57ab4cafab8` sits `detected` without an alert; both radar incident rows close only on a clean 07:00 weekday slot.
## [2026-10-29] update | Steel Browser fully removed from the estate — Camofox is the sole backend
- **Root cause of the long-standing `cloud_provider: steel` residue:** the 2026-07-30 rollback disabled the `browser-steel` plugin and set `browser.engine: camofox`, but left `browser.cloud_provider: steel` in `~/.hermes/config.yaml`. Every browser tool call then routed through `_instantiate_explicit_cloud_provider()` and failed with *"no registered browser plugin has that name: steel"*. Because `engine` was already correct, the split config was inert-but-wrong and survived ~3 months of audits.
- **Hermes (CT460):** `browser:` block now reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`, `cdp_url: ""`. The `browser-steel` plugin directory was **deleted**, and the `browser-steel` key was removed from both `plugins.disabled` and `plugins.entries`. `STEEL_BASE_URL` was dropped from `~/.hermes/.env`. **Hermes core source contains zero Steel references**, so there is no Steel fallback path in the tree — returning to Steel now means redeploying it.
- **CT201 (runtipi):** the entire Steel deployment was torn down — containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network` Docker network, the images `steel-browser-api:fixed` and both ghcr.io Steel images, and the project dir `/root/steel-browser` (~6 GB reclaimed; disk 70G → 64G used). The Tailscale Serve routes `/steel` (5173) and `/steel-api` (3000) were deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`. Nothing listens on 3000, 5173 or 9223.
- **CT333 (clawtest):** a **second, separate and orphaned** Steel install was found and removed — the `steel-browser` container (`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image. It had no Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on 127.0.0.1:9515), no reference from Nanobot's config or the CLI-anything browser harness; its last CDP navigation was 2026-10-20, it received no requests after 2026-10-21, and its page crashed 2026-10-22.
- **Estate-wide sweep:** every LXC on the PVE host was checked — no Steel containers, images, networks or listeners remain anywhere. Homarr, the homepage dashboard and the runtipi app configs had no Steel references.
- **Gateway restarted 01:21 AWST** to clear `STEEL_BASE_URL` from the inherited process environment; the new gateway tree (PID `3913841`) reads `STEEL_BASE_URL=0`, and Telegram, Discord, ntfy, email and Home Assistant all reconnected cleanly. The dashboard unit and `agentmail-webhook.service` still carry the variable in their inherited environment, but **no core code reads it**.
- **Reversibility preserved:** CT201's compose file and Dockerfile are archived under `/root/_archived-steel/`, plus `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. Backups: `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`, `.env.bak-pre-steel-cleanup-20261028T220947Z`.
- **Camofox verified healthy** after every change: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**, `/tabs` → `{"running":true,"tabs":[]}`, and live navigation confirmed working.
- **Pages corrected (15):** [[systems/browser-backend]] · [[decisions/rollback-steel-to-camofox]] · [[decisions/index]] · [[comparisons/steel-vs-camoufox]] · [[comparisons/camoufox-vs-chrome]] · [[infrastructure/docker-services]] · [[infrastructure/tailscale]] · [[infrastructure/hosts]] · [[systems/homepage-dashboard]] · [[concepts/browser-automation]] · [[runbooks/recover-docker-service]] · [[current-state]] · [[index]] · [[agents/nanobot]] · [[systems/scheduled-tasks]].
- **Corrects the 2026-10-29 00:3x audit entry**, which described Steel as merely "not reachable / endpoint dark". It is **removed**, not down. Per SCHEMA rule 9 the superseded claims are marked, not deleted.
- **Removed/archived:** none. **Secrets written:** none (variable names only; no key or URL values).
- **Unresolved:** the obsolete CT333 line in Nanobot's Obsidian vault (`Areas/Homelab/architecture.md`, "Steel Browser | Clawtest | http://localhost:3000") still stands — that vault is outside this wiki and was not modified. Also noted, unrelated to Steel: `Browser Use viewing unavailable: unsupported Hermes browser resolver` has logged in the gateway since 2026-10-11.
## [2026-10-29] stocktake | General Agent Estate Wiki maintenance — Tailscale table re-verified, and a 23-day-old "never re-authenticated" claim turned out to be false
- **Script:** `wiki-maintain.sh` ran clean (exit 0). `systems/skills-index.md` regenerated from the live skill tree — **302 skills in 69 categories**, with a new note that **45 skills** are parked under `~/.hermes/skills/.archive/` and excluded from the count. (303 `SKILL.md` files exist outside `.archive/`, but `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` is a nested alias duplicate of `productivity/obsidian-git-sync/SKILL.md`, so **302 unique skills** is the correct figure.) Freshness report written to `raw/stocktake/freshness-20261029.txt`; Quartz rebuilt (110 pages → 644 files) and deployed to `public/`.
- **Freshness report:** **2 stale / 105 fresh / 3 no-date** of 110 pages. The 2 stale were `incidents/2026-10-01-cliproxy-connection-auth-expired.md` and `incidents/2026-10-05-dead-model-pin-repin.md`, both at **23d**.
- **Updated [[infrastructure/tailscale]]** — State column rebuilt from `tailscale status --json`; **11 rows changed**. Two were **state transitions**: **`sftpgo` (CT299, pve) is back online** (the 2026-10-28 audit had recorded it `offline <1d`; `tailscale ping` → pong in 0s, `systemctl is-active sftpgo` → `active`, `tailscale ip -4` inside the guest → `100.88.223.70`), and **`m93p` is online despite a stale peer record** (`Online: false` in the JSON, but `tailscale ping` → pong in 1ms and SSH over the tailnet returned `m93p / up 11 days`) — the **second** occurrence of that pattern after 2026-10-26. The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged at **54** (53 live IPv4 peers + `hermes`); JSON reports **76 peers — 53 with a `100.` IPv4, 23 IPv6-only funnel ingress**. **No missing peers, no IP mismatches, CT601 still absent from both hypervisors.** Container identity re-confirmed from **inside** all 34 guests via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; the 3 stopped guests (CT209/CT255/CT555) and the 3 running-but-unconnected guests (CT105/CT1264/CT100) confirmed.
- **Updated [[incidents/2026-10-01-cliproxy-connection-auth-expired]]** — the residual claim *"the connection credential itself was never re-authenticated"* is **falsified**. A live `POST /v1/chat/completions` for `cliprox/terra` against `custom:Omniroute:20128` (CT600) returns **HTTP 200** — the connection has since been re-authenticated, independently of the incident's remediation. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`), so the two nodes carry separate credentials. Claim marked superseded rather than deleted (SCHEMA rule 9); the remediation itself (moving 3 jobs off the pin) was never reversed and remains correct.
- **Updated [[incidents/2026-10-05-dead-model-pin-repin]]** — every claim re-probed live: all 7 jobs are still on `custom:omni-101` with routing aliases (`free-stack` ×6, `smart-route` ×1) ✔; both aliases return **HTTP 200** ✔; `ollama-cloud/qwen3.5:397b` still **HTTP 410** (retired 2026-09-25) ✔; `nvidia/deepseek-v4-flash` still **HTTP 400** (absent from the live catalog) ✔; `command-code/gpt-5.6-terra` now returns **HTTP 401** (credential) rather than the original **403** (plan-gated) — the pin is still dead, only the failure signature changed. Frontmatter `updated`/`verified_on` bumped to 2026-10-29 with `created` left at 2026-10-06 as the data period.
- **Updated [[systems/scheduled-tasks]]** — the Known-Issues entry repeating the "never re-authenticated" claim now carries the correction. The page's fleet figures were re-checked against live `jobs.json` and remain accurate: **29 jobs / 26 enabled / 3 disabled**, 5 non-clean `last_status`.
- **Verified, no change needed:** [[systems/homepage-dashboard]] (CT201 `ss -ltn` shows 8084 and 8085 listening, **no 8082** listener, Homarr on 8102 — all as the page claims); CT201 Serve routes (`/` → 8085, `/mgr` → 8084 on **both** `:443` and `:8443`, exactly as documented, Steel routes still gone); CT460 Serve routes (12 paths on `:443` + `:8443/webhook` Funnel + `:9443`, matching the page).
- **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed.
- **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values).
- **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died
silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated
`wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key;
(B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401**
with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent
call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq`
→ **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL →
**2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard
restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and
08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free.
- **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference,
and `verified_on: 2026-10-30`.
- **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table +
why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second
*superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`).
- **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
*superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar
rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues.
- **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed →
delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**,
`missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed.
- **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and
**[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12
(3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`,
`dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed`
at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed.
- **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors.
- **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since
2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**;
non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak`
reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged.
- **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap.
- **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
key or token value.
## 2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state
- **Ran `wiki-maintain.sh`.** Regenerated `systems/skills-index.md` (**302 skills / 69 categories** — unchanged; the
generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote
`raw/stocktake/freshness-20261030.txt` (**0 stale / 109 fresh / 3 no-date of 112 pages**), rebuilt Quartz
(654 files) and deployed to `public/`.
- **Freshness is not accuracy.** The age scan found nothing, but a live-vs-page diff against the running estate found
real drift on five pages. All corrections below were verified against live sources before being written.
- **[[infrastructure/tailscale]] — node table rebuilt.** `tailscale status --json` now reports **50 non-funnel nodes**
(49 live IPv4 peers + `hermes`), not the 54 the page claimed. **2 IP changes** (`anthonys-mac-mini` 100.74.17.15 →
100.76.60.37; `omarchymac` 100.84.180.69 → 100.126.173.115 — the peer formerly listed as `omarchymac-1`) and **4 nodes
dropped out of the tailnet entirely** (`anthonys-imac`, `cachyos`, `omarchymac-1`, `omni`). Every offline age refreshed.
`m93p` reads `Online: false` for the **third** consecutive audit yet is plainly up (`tailscale ping` → pong, SSH → `up 12
days`), so the row records it online. Total peers **72** (49 IPv4 + 23 IPv6-only funnel). CT460's `:443` route set
matched the page exactly; CT201 serves only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`.
- **[[systems/scheduled-tasks]] — fleet ledger corrected.** Live `jobs.json` shows **29 jobs (25 enabled / 4 disabled)**, not
the 28/25/3 on the page: `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused 2026-10-30 01:04:22** after its
00:01:08 slot closed **`unknown`** on a gateway-connection failure. Non-clean `last_status` is **4** (not 3), and the
standing "every `failure_streak` reads 0" claim was false (`ea79a2107e36` = 1). "Failing (delivery only)" is **2**, not 1
(`fad5a86979d4` joined `066bb5191280`). The retired DHL row was dropped from the job table (it lives in Removed Jobs).
Seven job rows carried **stale `error` claims** for jobs that have since cleared to `ok` — each was marked superseded
with a dated correction rather than rewritten (SCHEMA rule 9). The `executions.db` retention floor note was corrected to
the live **2026-10-24 09:22:36**. Window tallies: **2026-10-29 = 177 / 0** and **2026-10-28 = 169 / 2** (both +1 on the
page — the 23:59 Wiki Audit row had still been `running` at audit time).
- **New incident page: [[incidents/2026-10-30-mnemosyne-health-monitor-paused]].** The Mnemosyne Health Monitor's own
failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) —
`deliveries.db` holds **zero rows for 2026-10-30**, so the incident row is the only trace. The job was **not re-enabled**
and the gateway was **not restarted** by this run; root cause is **unverified** and recorded as such.
- **[[infrastructure/hosts]] — four corrections.** CT201's LAN address is `192.168.178.77`, not `192.168.178.166`; CT450's Hermes Agent is **v0.22.5** (2026.9.24),
local **`bf0203a794a0`** — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both
(a stale *root-level* `/usr/local/bin/hermes` on CT450 still reports v0.19.0 and is **not** the live agent —
the live one is the `hermes`-user install under `~/.hermes/hermes-agent`); the cron count is now 29/25/4; and
the orphaned `omni`
peer has **expired out of the tailnet entirely** (no peer named `omni`, no peer at 100.93.204.4), so the "offline 69d"
description no longer applied. The **skills count of 302 was left alone** — it is correct.
- **[[systems/homepage-dashboard]] — five corrections, and a second pass to fix my own first pass.**
"All Apps" is **114 → 123 live apps**, not 122 (the API dedupes by `(name, normalised href)` across groups;
the literal `All Apps` group in `services.yaml` holds 122 and is not what the tab bar counts). The grid is a
responsive `repeat(auto-fill, minmax(230px, 1fr))` (served `/dash.css` line 3640), not a fixed 5-column grid.
The tab list was wrong — but **my first correction was wrong too** and had to be redone: the live tab bar is
built from the service *groups* (`renderTabs()` iterates `state.groups`), **not** from the `layout:` block in
`settings.yaml`. That block and its `tab:`/`columns:` keys are **inert** — `grep -c "Overview" /dash.js` = 0 and
`grep -c "columns" /dash.js` = 0. There is **no "Overview" tab**; the live tabs are `☆ Pinned`, `All (123)` and
the **14 groups** (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20,
Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved
theme is **`{"pair":"Boarding Pass","mode":"light"}`** — one of **59** theme pairs; its light-mode amber is
`#795000`, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. **`manager.py` runs inside
the `homepage-manager` container** (its container `Cmd` is the `pip install … && python3 /opt/homepage/manager.py`
line, confirmed via `docker inspect` and `docker top`); only **`asset_proxy.py`** is a host process
(`0.0.0.0:8085 users:(("python3",pid=4038))`). And the "no Traefik container exists" evidence was wrong — the
`runtipi-reverse-proxy` Traefik container exists but is **stopped** (`Exited (255) 2 months ago`); the *conclusion*
that Traefik does not occupy 443 stands. The same evidence error was corrected in [[infrastructure/tailscale]].
- **[[current-state]]** and **[[index]]** updated with the new incident and the corrected fleet numbers.
- **Verified, not assumed:** CT600 **running** and CT601 **absent from both hypervisors** (`pct config 601` → no such
config on `pve` or `m93p`); the CT201 `:443`/`:8443` Serve tables; CT201 listeners 5173/3000/8082 (**none** — Steel's
removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row
(read from `jobs.json`, `executions.db` — whose `cron_incidents` table lives inside it, the standalone
`cron_incidents.db`/`incidents.db` files are empty stubs — and `deliveries.db`).
- **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
the steel-removal records across the wiki (confirmed still true).
- **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value.
## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
recorded wrong.**
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**;
PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22.
The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**.
Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled.
- **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
`last_status`, before writing an incident.**
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01).
- **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`.
- **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.
## [2026-11-01] audit | Nightly review — Conduit 0.122.2, the Local Live fault, a force-drained reload, dead Asana paths
- **NEW PAGE — [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].** The app's **"local" voice mode
never applies** even though its connection test succeeds — because the test checks **socket reachability only**.
Every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint →
**HTTP 501**, *"start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**
across 25 observed turns, and its realtime socket **accepts the connection but rejects `session.update` as an
unknown event**. `voice.mode` is a **phone-side** selector: **no `mode` key exists** in `config.yaml → voice`
or `conduit-settings.json` (whose only device state is `devices/<UUID>/mode: small` ×3), so "the change did not
apply" cannot be diagnosed on host config. Evidence from session `20261031_193709_40b9c2` (started **19:37:09**,
30 msgs / 17 tool calls, still open at the 00:2x reading); `voice.local_live.front_model` was repointed to
`mistral/voxtral-small-latest` mid-session (`config.yaml` mtime **21:02:30**). **Nothing was changed on the
Mac.** ⚠ One misread was caught in-session and is recorded as a lesson: `local_live.py:882` is the **only**
writer of `_local_live_last_request`, inside the chat-completions route — so a "recent `last_request_at`" was
the investigation's **own probe**, not phone traffic (`last_request_at` is `null`).
- **NEW PAGE — [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].** The **18:06:41** gateway reload
waited the **full 1,800 s** drain cap and force-drained at **18:37:38** (new gateway **18:38:43**), orphaning
`ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot*, **6 h 38 m 09 s** in) which closed **`failed`** at
**18:39:15** on *"lost its durable fire claim ownership"*. Derivative watchdog reds at 18:39:23 / 18:45:23 /
19:00:22 cleared **19:15:21**. A **second, clean** restart followed at **19:23:39 → 19:24:40**. Same class as
the 10-26 planned-restart orphan — **second instance, first on a plain reload**.
- **NEW PAGE — [[incidents/2026-10-31-asana-access-failures]].** Both Asana paths on CT460 are non-functional:
the MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never
authenticated** (`~/.hermes/mcp-tokens/` holds only `asana.client.json`, **319 B**, mtime 2026-10-26 01:07;
logs read *"MCP OAuth setup failed for 'asana' … no cached tokens found. Run `hermes mcp login asana`
interactively first"*), and the hosted connector rejects the call shape. The repair needs an **interactive**
login, which cron cannot perform.
- **UPDATED — `systems/conduit.md`** (13,184 → 18,722 B; frontmatter `updated` / `verified_on` → 2026-11-01).
Version `0.69.2` / `942ddb8` → **`0.122.2` / `9eac857`** (10-26 reading kept explicitly superseded); the
⚠ *"uncommitted patch, five releases behind"* warning is **RESOLVED** — the tree is clean and
**`MEMORY_TIMEOUT_S = 10.0` ships upstream** in `dashboard/api/memory.py` (with a `wait_for` backstop and
`tests/test_memory.py`), so the 10-23 in-tree edit is no longer needed; dashboard gateway
`pid 4042003` (10-29 08:03:26) → **`pid 176771` (10-31 19:35:02)**; the **call-store shape change** is recorded
(`v: 1`; **`history` empty**; **`last_request_at: null`**; `ends` = 2 × `outcome: stopped` at 21:06:05 /
21:10:48; `settings.quiet_at_screen: true` = #479); data volume 930.0 MB → **938.2 MB** (983,793,664 B) with
**2,623,879 `events` / 4,384 `buckets` / 504 `windows`** (≈8 MB/day, no retention policy); new dated section;
`Open items` — patch item marked resolved, three new local-live items added.
- **UPDATED — `systems/scheduled-tasks.md`** (same pass): fleet **29 jobs (26 enabled / 3 disabled)**; 2026-10-31
closed **160 / 4**; non-clean `last_status` **4** (2 `error` + 2 `delivery_failed`); failing jobs **1**
(`ea79a2107e36`, `failure_streak` 2); superseded readings marked rather than overwritten.
- **UPDATED — `systems/hermes-agent.md`:** Asana MCP auth-state re-verification note (enabled but never
authenticated) + both 2026-10-31 gateway restarts (18:06:41 force-drain, 19:23:39 → 19:24:40 clean).
- **UPDATED — `index.md`:** frontmatter and banner → **2026-11-01**; Conduit entry (version, patch resolved,
gateway pid, new fault, empty call history); **three new incident entries** added above the 10-30 pair.
- **UPDATED — `log.md`:** this entry.
- **Checked, unchanged:** ship rows before 10-30 (nothing new); `inc_20261031_000001` (gateway-update) is **still
open**, `inc_20261030_000001` closed 2026-10-30 18:05; **3** `deliveries.db` rows for 2026-10-31 carry a
failure error (`child run failed: exit status 1`) — they are the failing runs' own notices, not new faults;
no page archived; **no secrets, tokens, passwords or keys written**.
- **Probes (context, not findings):** `omniroute-1` `100.78.63.20:20128` → **HTTP 200**; `m93p` `100.104.51.98`
and `pve` `100.96.100.82` answered neither ICMP, SSH nor `:8006` from CT460 at the 00:2x reading —
recorded as **unverified reachability from this container**, not as a host outage.
- **Mirror catch-up (a whole-tree pass, beyond this audit's own pages).** Because the previous copy bridge only
ever synced the pages an audit touched, the user-facing vault had silently drifted. This pass re-copied **every
committed page** in the authoring tree into **both** vault layouts (`/home/hermes/wiki/agent-estate-wiki/` and
`.../wiki/`) — **61 pages** — which closed the 9-page drift and picked up everything the **2026-10-31** audit had
changed and never mirrored (`current-state.md`, `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md`,
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`, `infrastructure/hosts.md`, `systems/mnemosyne.md`)
plus 53 older stragglers. **Deliberately left unsynced:** the **6** files the 2026-10-31 03:00 stocktake has left
**uncommitted** in the authoring repo (`infrastructure/docker-services.md`, `infrastructure/proxmox.md`,
`infrastructure/tailscale.md`, `systems/messaging-integrations.md`, `systems/omniroute.md`,
`systems/skills-index.md`) and untracked `raw/stocktake/freshness-20261031.txt` — the mirror convention is
*"(authoring COMMIT_SHA)"*, so uncommitted content stays out until the stocktake's own commit lands.
## [2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago
- **BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake).** `wiki-maintain.sh` exited **1** at
step 3/3 with an esbuild panic. Root cause was **not** esbuild: `incidents/2026-10-31-conduit-voice-local-mode-not-applying.md`
had an **unquoted YAML `title:`** containing the substring `: ` (from `"voice mode: local"`), so YAML parsed it as a
mapping entry and Quartz aborted with `bad indentation of a mapping entry (2:27)`. A scan of all **117** pages
found **exactly one** offending frontmatter line; the title is now double-quoted with escaped inner quotes.
**This is a latent trap worth remembering:** any future page title containing `": "` or `":` unquoted will
silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather
than a YAML error.
- **STALE PAGE — `incidents/2026-10-09-memory-recall-relevance.md` (23d) — RESOLVED, not stale.** The page's
central claim was that the recall-ranking fix *"was reported saved and is **not** present in live config"*, and
that conflict `152` was *"the only open conflict"*. **Both claims are false, and the page was the reason the
incident looked open for three weeks.** Live `~/.hermes/mnemosyne/config.yaml` reads `vec_weight: 0.5`,
`fts_weight: 0.3`, `importance_weight: 0.3` (plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) —
**the weights live in Mnemosyne's own config file, not in `~/.hermes/config.yaml`'s `memory.mnemosyne` block
that the 2026-10-10 probe read.** The 2026-10-10 verification probed the wrong file and concluded "not applied".
**The original regression test now passes:** `mnemosyne recall "Are you up to date"` returns only substantive
current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode
(`id 32f469aae09d56dc`, still in the DB) **no longer outranks anything**. Conflict `152` was auto-resolved
**2026-10-29 23:53:47Z** ("low-confidence LLM extraction noise") — it has not been open since. All three open
items closed; page moved to `status: resolved`. Live tier counts at 2026-11-01: working **941**, episodic
**1,124**, canonical **90**, conflicts **175** with **11 open** (ids 165–175, all one-batch extraction noise
sharing a single `fact_a_id`), working embeddings **927/941 (98.5 %)**, episodic **1,124/1,124 (100 %)** — both
above the 95 % floor, no `reindex` needed.
- **UPDATED — `systems/mnemosyne.md`:** stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01
block added, including the weights' actual home and the resolved-incident pointer.
- **UPDATED — `infrastructure/tailscale.md`:** table rebuilt from live `tailscale status --json` — **1 state
transition and 5 offline-age drifts, no IP changes, no missing peers**. `omarchymac` (100.126.173.115) went
**offline** (`Online: false`, `LastSeen 2026-10-31 13:20Z`, `tailscale ping` times out — a genuine transition;
the JSON `OS` field reads `linux`). `m93p`'s peer record reads `Online: false` **again** — the **4th
consecutive** stale record — while `tailscale ping` → `pong` and SSH returned `up 1 day, 12:02`, so the row stays
**online**. Ages refreshed: cloudreve 98→**99d**, freshrss-new 74→**75d**, omarchybook 9→**10d**, openmausbot
9→**10d**, ubuntu-server-xfce 71→**72d**; `conduit` (55d) and `elementsynapse` (102d) were already correct.
- **UPDATED — `infrastructure/docker-services.md`:** **CT201's Tailscale IP was wrong** — the page read
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which
`[[infrastructure/hosts]]` already had right). The CT200 inventory had **three misspelled `overtchat-*`
entries** (written `overchat-*`) and **one stale entry**, `worldmonitor-seeder-fast`, which is not running —
the list is now regenerated verbatim from live `docker ps`. **CT333's first container was renamed**:
`adoring_jemison` → **`thirsty_payne`**. Counts re-verified: CT200 **61**, CT201 **20**, CT333 **8**,
CT1001 **1** (`sshwifty`), CT460 `ignis` running.
- **UPDATED — `infrastructure/proxmox.md`:** storage percentages refreshed against live `pvesm status` on **both**
hosts (`pve` `fred-qnap` 82.45→**82.77 %**, `SSD1` 63.80→**64.32 %**, `local-lvm` 13.60→**13.75 %**; `m93p`
`nfs-proxmox-*` 82.45→**82.78 %**, `omb-internal` 58.00→**57.92 %**). `pct list` / `qm list` on both hosts
re-confirmed — **no CT601 on either hypervisor**, CT255 still `running`, VM 103 `cachyos-m93p` stopped, VM 106
`openmausbot-vm` running. `m93p` uptime recorded as **up 1 day, 12:02** with a **load average of
7.66 / 12.80 / 11.32** — elevated, worth watching.
- **UPDATED — `systems/omniroute.md` / `systems/messaging-integrations.md`:** re-probed today — both OmniRoute
nodes answer **401** on `/v1/models` (alive, auth required), CT600 and CT101 both `running`; Mattermost's
`systemctl is-active` → `active`, `LISTEN *:8065`, `/api/v4/system/ping` → **HTTP 200**. Frontmatter
`verified_on` bumped to 2026-11-01.
- **REGENERATED — `systems/skills-index.md`:** **301 unique skills** in 68 categories, **46 archived**. The count
is *unique `(category, skill_name)`* pairs, not `SKILL.md` file count — live `find` returns **302** files, and
the one difference is the known nested alias duplicate
`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` vs `productivity/obsidian-git-sync/SKILL.md`.
**301 is correct — do not "fix" it to 302.** One skill was archived since the last run:
`hermes-runtime-worker-diagnostics` (46 archived, up from 45).
- **FRESHNESS:** the 2026-11-01 report lists **1 stale / 113 fresh / 3 no-date** of 117 pages. The single stale
page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after
the fixes, per the script → fix → re-run → commit order.
- **Probes (context, not findings):** `m93p` SSH over the tailnet timed out on the first two attempts and
succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). `homarr` on
CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's
`:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as
the homepage-dashboard page records. No secrets, tokens, passwords or keys written.
## [2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by `-15`, and a red status with no row behind it
**Window:** 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on `state.db` (sessions),
`~/.hermes/cron/{jobs.json,executions.db,deliveries.db}`, `gateway.log` / `agent.log.1`, and both git trees.
- **NEW — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`** (`status: active`):
the Sunday 18:00 `79f9bb41ecf5` closed `unknown` at 18:05:13 with *"Gateway connection failed; no local
fallback"*, **delivered nothing**, and its incident row `79f9bb_9c443b602eec` is **still `alerted`**.
`deliveries.db` has **no row after 11:01:12**. Batch-mate `b63ed5817212` hit the identical error and
self-healed in 68 s. The same page records the **inverse** silent failure: `ecc1fc13655a` reads
`error`, `last_run 11-01T12:01:00`, `failure_streak: 1` — with **no execution row and no incident row**.
- **NEW — `incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md`** (`status: resolved`): three
workers `SIGTERM`ed (`-15`) after adopting their claim (`85edf6772bca` 07:30:46; `96eb95ce12f2` +
`0e753e9720fd` 09:02:18) → *"whether side effects ran is unknown"*. **Fourth variant of the orphan
class, and the first with no gateway restart/reload/drain anywhere in the window** (PID `175269` up
since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a
**30 s worker-ack window vs a ~70 s cold start** (`Plugin discovery complete: 83 found, 69 enabled`).
All three incident rows `resolved`; one execution left `unknown`.
- **UPDATED — `systems/scheduled-tasks.md`**: `## Count` advanced to the 2026-11-02 window (**167
`completed` / 1 `running` / 2 `unknown`** vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the
first non-clean day in four); fleet **29 jobs / 26 enabled / 3 disabled** unchanged; **unclosed
incident rows 1 → 1 but a different row** (all three rows the 10-31 audit left open closed on 11-01,
including **`0e753e_e7f8194a7d06`**, open since **2026-10-01** — 31 days); new `## Re-verified
2026-11-02` section covering the three failure modes, the 09:00 cold-start pair and the ledger.
**Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`).
- **UPDATED — `systems/messaging-integrations.md`**: Telegram degraded all day (dual-stack
`api.telegram.org` `ConnectTimeout` → IPv4-literal fallback; disconnect 20:51:53, polling restarts
22:33:30 / 23:12:23), and `getent hosts` currently returns **only an AAAA record** for
`api.telegram.org` — consistent with, but not proof of, a systematic timeout on the path DNS hands
the client. The **`relay` platform has had no adapter for ~29 h** (`Reconnecting relay` attempt
**147 @10-31 18:39:24 → 336 @11-02 00:10:47**, counter never reset — itself the proof of no gateway
restart). `066bb5191280`'s `thread_id 34795` home-DM miss is the **second confirmed variant** of the
thread-mismatch bug: the thread-less fallback fires, the content lands, and the **run is recorded
`failed` anyway**. Day's delivery health: **8 rows / 7 `delivered` / 1 `failed`**.
- **UPDATED — `index.md`**: `updated`/`verified_on` → 2026-11-02, the audit summary line rewritten for
this window, and the two new incident links added to the incident list.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned;
no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts
below instead of a bulk copy) — an audit's mandate is smaller than a full sync.
- **DRIFT (context, not findings):** the authoring tree was clean and in sync with Gitea
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
## [2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see)
- **Scope:** `state.db` sessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST; `cron/executions.db`,
`cron/deliveries.db`, `cron_incidents`, `cron/jobs.json` (all read-only). Authoring tree
`/home/hermes/workspace/agent-estate-wiki` was **clean and in sync with Gitea** (`origin/master..HEAD`
empty) before this audit; `systems/skills-index.md` and `raw/stocktake/freshness-20261102.txt` were
**committed on the stocktake's behalf** (see below).
- **NEW — `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md`:** the **third iteration-cap
in seven days** (10-27 tool-call budget, 10-28 cap, 11-02 cap) on `dca8482e4f76`
(**07:00:33 → 16:01:52, 9 h 01 m 19 s**), and **the first that raised no `cron_incidents` row at all** —
it closed `completed` / `error: NULL` / `delivery_outcome: suppressed`. Both Stage 2 slots exited
`[SILENT]`, so **nothing reached Notion, no cover letters were written and the 164 deduped candidates
were never persisted**. Page carries the **"green row, no artefact" detection table** contrasting the
three occurrences, and records the mid-run **URL-canonicalisation fix** (`au.seek.com` vs
`au.seek.com.au`) that had been silently dropping SEEK and Indeed entries from dedup.
- **UPDATED — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`:** status
**`active` → `resolved`**. Live evidence: the **same execution row**
(`be6ec11a8e68428084af9ab18fb148db`) closed `completed` / `delivered` at **01:27:22** after
**7 h 26 m 43 s**, delivery at 01:27:21.802246, incident `79f9bb_9c443b602eec` auto-closed at
**01:27:21.606691**. The 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**,
not a verdict. Resolution section added; open questions 1–3 answered, **question 4 retained** (the two
unreconciled admission files `141448f9…json` / `69b5588f…json`).
- **UPDATED — `systems/scheduled-tasks.md`:** `## Count` advanced to the 2026-11-02 window —
**177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) vs 167/1/2 on 11-01, 160/4 on
10-31, 138/0 on 10-30; fleet unchanged at **29 jobs / 26 enabled / 3 disabled**; **unclosed incident
rows 1 → 0** (the first zero in the period); both 11-01 `unknown` rows resolved and `ecc1fc13655a`'s
ghost `error` cleared; deliveries **9 rows / 9 `delivered`**. New **`## Re-verified 2026-11-03`**
section documents the two invisible failures, the stocktake artefacts committed, the
`substr(started_at,1,10)` grouping rule and the **"do not read `running`/`unknown` as a verdict before
the expected runtime has elapsed"** correction. Frontmatter `updated`/`verified_on` → 2026-11-03,
`review_after` → 2026-11-10.
- **UPDATED — `systems/browser-backend.md`:** `updated`/`verified_on` → 2026-11-03 plus a `## History`
entry for the **2026-11-02 REST-server wedge** (requests timing out with nothing reaching the service
log, cleared by `systemctl --user restart camofox-browser.service`; the four-board Job Radar run then
proceeded). **Live re-probe 2026-11-03 00:16:14 AWST:** unit `active`, `MainPID 953299`,
`ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`, `/health` → **200**
`{"ok":true,"engine":"camoufox","browserConnected":false,…}`, `/tabs` → `{"running":true,"tabs":[]}`.
Page now states explicitly that this wedge is a **different fault from the 2026-10-12 memory ceiling**
(hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control.
- **UPDATED — `index.md`:** `updated`/`verified_on` → 2026-11-03, the audit summary line rewritten for
this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated
**✅ RESOLVED 2026-11-02T01:27:21**.
- **COMMITTED on the stocktake's behalf:** `1a8e90ea2271` finished with `systems/skills-index.md`
modified and `raw/stocktake/freshness-20261102.txt` untracked. Both are valid artefacts — freshness
**4 stale / 113 fresh** at a >21-day threshold, skills-index regenerated with **Devops 108 → 109**
(`pulse-turn-handling`) — so this audit committed them rather than let the next slot overwrite them.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a
bulk copy).
- **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
id, PID, thread id and timestamp above is a non-sensitive operational identifier.
## [2026-11-04] audit — window 2026-11-03
**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` /
1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since
10-31**.
**New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1
(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on
**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35,
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at
**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident
`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still
open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse
carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream
active timeout"*).
**Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the
recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery
tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended
`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean
state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`.
Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.**
**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16,
14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
**New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author
*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`;
`config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the
replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory
remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched,
still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
blocked and had to be rephrased.
**Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:**
`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows,
7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39).
⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 =
**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts
uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this
audit committed them.
**Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop`
`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the
2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02).
**Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
embedded credential was seen during verification and is not reproduced here ([REDACTED]).
## [2026-11-05] audit - window 2026-11-04
- **New page:** `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` - two 07:00 jobs
(`dca8482e4f76` **Daily Marketing Job Radar - Discovery**, `85edf6772bca` **Tech & AI Newsletter Digest
(Analytical)**) died **36 s apart** (09:41:44 / 09:42:20) on `RuntimeError: Upstream stream failed before
completion.` after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`; both incidents still open
(`state = alerted`). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own
candidate files, re-verified every candidate live, and wrote **5 roles (board 479 -> 484) + 1 canonical
cover letter** (`MISSING_COVER_LETTER_COUNT 0`), consuming the handoff at 16:08:59.
- **Updated `incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md`** - CT299 was frozen **twice**, not
once. Second freeze: a **manual** `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged
(`unexpected status`) until **11-05 ~00:30**. Also records the earlier clean failure at **07:36:44**
(log `vzdump-lxc-299-2026_11_04-07_36_44.log` on the dump target) and three `push_file` tasks from CT460
at 07:17-07:33, plus the abandoned `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`. Live probe 2026-11-05
00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs.
- **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes.
- **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` /
2 `failed` / 1 `running`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**;
deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third
day (5 h 41 m 46 s, `free-stack` pin).
- **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to
`CL_MODEL=deepseek-paid-smart` (line 82); the 11-04 run pinned `gemini/gemini-3.5-flash` after **HTTP 402**.
`grep` shows **no `CL_MODEL` in `~/.hermes/.env`** - the override was process-local and did not persist, so
the next run inherits the default again. Endpoint probe 2026-11-05: `omniroute:20128/v1/models` -> **401**
and `100.78.63.20:20128/v1/models` -> **401** (live, key-gated) - so the route is reachable; the same model
serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested.
- **Updated [[current-state]]** - new "Recent Changes (2026-11-05)" block; the page had been sitting at the
2026-10-30 block across five audits.
- **Updated [[index]]** - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved
2026-11-03 -> 2026-11-05.
- **Live verification this audit:** pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup
freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from
CT460 on `100.88.223.70:8080`; `jobs.json`, `executions.db`, `cron_incidents`, `deliveries.db` read live.
- **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or re-pinned; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
- *network-map.md re-verified (live probe 2026-11-05):* corrected the ct460 port table — **9377 is Camofox browser REST** (`/home/hermes/services/camofox-browser`), not Stratos; `8790` and `5174` are no longer listening; added `9120` (wiki serve), `9222` (Camofox CDP/WS), `7777` (OpenChamber), `6767` (Paseo), `8085` (AgentMail webhook), `9900` (gateway). Added the full Tailscale Serve table (12 routes on `:443`, `/` on `:9443` → 5173, Funnel `:8443` `/webhook` → 8085). Two routes are dead: `/agent-estate` → 8769 and `/synergy-prep` → 9123 have no listener.
## [2026-11-05] correction + live probe | fleet count, 11-04 ledger composition, `deepseek-paid-smart` dispatch
- **What was wrong.** The 2026-11-05 nightly-audit entry (written 00:5x AWST) recorded the fleet as **158 jobs
(144 enabled / 14 disabled)** and the 11-04 ledger as closing **1 `item_suppressed`**. Both were wrong. Live
re-reads - `~/.hermes/cron/jobs.json`, `hermes cron list`, and the in-flight job count - each give **29 jobs /
26 enabled / 3 disabled**; `executions.db` shows 11-04 at **170 rows - 167 `completed` / 2 `failed` / 1
`running`** (this audit), with **no `item_suppressed` status anywhere on 11-04**.
- **Where corrected:** `systems/scheduled-tasks.md` (Count bullet + correction clause),
`current-state.md` (11-05 block), `index.md` (11-05 ledger paragraph), and this entry. No other page carried
the bad number.
- **11-03 re-settle:** `171 completed / 1 failed` (172 rows) - its `running` row has since completed - so 11-03
and 11-04 are the only consecutive days with a `failed` row since 10-31.
- **Live probe - model dispatch, 2026-11-05 01:33:12 AWST (first-hand, keyed):**
`POST http://omniroute:20128/v1/chat/completions` with `deepseek-paid-smart` -> **HTTP 503**
`ALL_TARGETS_SKIPPED` (`poolSize: 5`, `attempted: 0`). The same request to
`http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, `model:
deepseek/deepseek-v4-flash`. Unkeyed `/v1/models` on both hosts returns **401** - auth-gating, not health. The
fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101`, i.e.
the `omniroute-1` node), which is why the audit still completes while a direct CT600 probe 503s.
- **Re-verified and matching (no change needed):** `cron_incidents` - 3 unclosed rows, all `state = alerted`;
`deliveries.db` - 8 rows on 11-04, **8/8 `delivered`**, including both `for_failure` notices (09:41:43,
09:42:20); non-clean `last_status` **3**; pin distribution **14 `smart-route` / 6 `free-stack` / 8 unpinned /
1 `deepseek-paid-smart`**.
- **Files touched this pass:** `systems/scheduled-tasks.md`, `current-state.md`, `runbooks/job-radar.md`
(`CL_MODEL` caveat now carries the keyed probe), `index.md`, `log.md`. Frontmatter `updated` / `verified_on`
bumped to 2026-11-05 where stale.
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted** - the API key was read from
`~/.hermes/.env` into the shell for the probe only.
## [2026-11-05] stocktake | Skills-index generator divergence FIXED at the root; Docker + Tailscale drift corrected against live probes
- **Ran `wiki-maintain.sh`** (repo canonical path) - exit 0. Regenerated `systems/skills-index.md`
(**302 skills / 68 categories**, `.archive/` excluded, 46 archived), wrote
`raw/stocktake/freshness-20261105.txt` (**0 stale / 122 fresh / 3 no-date of 125 pages**), rebuilt
Quartz, deployed to `public/`, restarted `hermes-wiki-serve.service`.
- **ROOT CAUSE FOUND AND FIXED - the `systems/skills-index.md` flip-flop had TWO WRITERS.** The repo
`wiki-maintain.sh` (mtime 2026-11-03) and `~/.hermes/scripts/wiki-maintain.sh` (mtime **2026-08-17**) are
**different scripts**, each with its own generator: the repo one scans with `find` excluding
`*/.archive/*` (-> **302 skills / 68 categories**); the stale one uses `Path.rglob('SKILL.md')` and a
2-segment category heuristic (-> **349 skills / 26 categories**, because it counts the parked archive).
`wiki-maintenance/SKILL.md` line 569 pointed the **23:59 nightly audit** at the stale path while the
**03:00 stocktake** ran the repo path - so each job overwrote the other's page and the wiki oscillated
nightly with **no error raised by either job**.
- **Fix:** `~/.hermes/scripts/wiki-maintain.sh` is now a **delegating wrapper** with **no generator logic**
- it `exec`s the repo script. The repo script gained **`--no-build`** so the wrapper honours the
interface its callers were written against without carrying a second copy of the generator.
- **Verified by execution, not inspection:** `bash ~/.hermes/scripts/wiki-maintain.sh --no-build` ->
`Written ... with 302 skills in 68 categories`; both scripts pass `bash -n`.
- **SKILL.md** repointed at the repo path, with a note that a generator must live in exactly one place.
-> [[incidents/2026-11-05-skills-index-generator-divergence]]; the "still open" item on
[[incidents/2026-08-15-skills-index-drift]] is closed.
- **`infrastructure/docker-services.md` corrected against live `docker ps` (all hosts, 2026-11-05 05:2x):**
- CT200 **61 running / 68 total** (was "62"); the **`worldmonitor-seeder-scheduled`** entry recorded on
2026-11-04 as "unhealthy" **does not exist on any host** - `docker ps -a --filter name=seeder` across
CT200/201/333/460 returns **zero** matches. Live CT200 shows exactly three worldmonitor containers
(`worldmonitor` `Up 5 days` **healthy**, `worldmonitor-redis`, `worldmonitor-redis-rest`). There is **no
seeder container and no worldmonitor-seeder cron job** anywhere in the estate.
- CT333 first container renamed again: `nostalgic_leavitt` -> **`flamboyant_mahavira`** (image
`ghcr.io/github/github-mcp-server`, created 2026-11-04T10:53:53Z). CT333 = **8 running / 9 total**.
- CT201 **20 running / 26 total**; CT460 **1 running / 1 total** (`ignis` `Up 5 days`); CT1001
**1 running / 9 total** (`sshwifty`). Per-host "Last verified 2026-11-05" added.
- **`infrastructure/tailscale.md` re-verified against live `tailscale status --json`:** the node table is
**exact** - 52 rows, 52 live IPv4 peers, **no live peer missing from the table, no table row without a live
peer, and every Online/offline state matches** (checked programmatically, not by eye). Two offline ages
corrected: `conduit` 59d -> **60d**, `ubuntu-server-xfce` 75d -> **76d**. `tailscale serve status` still
matches the recorded table exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443`
tailnet route).
- **`infrastructure/hosts.md`:** skills figure corrected to **302 skills / 68 categories** (was "69
categories"); `updated` / `verified_on` -> 2026-11-05.
- **`index.md`:** skills-index bullet refreshed to the 2026-11-05 figures (302 / **68 categories**,
freshness **0 stale / 122 fresh / 3 no-date of 125**), the `_archive/` typo corrected to `.archive/`, and
the new incident linked.
- **Freshness:** the 8 pages stale on 2026-11-04 were all re-verified and refreshed, so the 2026-11-05 report
reads **0 stale**. The 3 pages with no `updated` date are `SCHEMA.md`, `log.md` and
`experiments/active/example.md` - `log.md` is append-only by design, the other two are templates.
- **Files touched this pass:** `systems/skills-index.md`, `infrastructure/docker-services.md`,
`infrastructure/tailscale.md`, `infrastructure/hosts.md`, `index.md`,
`incidents/2026-11-05-skills-index-generator-divergence.md` (new),
`incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`.
- **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
## [2026-11-06] audit - window 2026-11-05
- **Window reviewed:** 2026-11-05 00:00-24:00 AWST; live reads 2026-11-06 00:1x-01:0x AWST. Fleet unchanged at
**29 jobs (26 enabled / 3 disabled)** - a fourth consecutive audit at the same count, no job added, retired,
disabled or re-pinned.
- **New page: [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (status: OPEN).** Three live
`.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in
plaintext in **17 rows of `~/.hermes/mnemosyne/data/mnemosyne.db`** - `memories.content` 4, `episodic_memory.content`
3, `working_memory.content` 1, `gists.text` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - and
`CAMOFOX_API_KEY` sits verbatim in the `prompt` of two **enabled** Facebook Birthday jobs (`0e753e9720fd`,
`fad5a86979d4`) and hence in `cron/deliveries.db`. Method: exact-substring match of each `.env` value across
**every TEXT/BLOB column of every table** in `mnemosyne.db` plus its FTS mirrors; byte-membership scans of the
10 `.db` files under `~/.hermes/config-backup/` (six are 0-byte; only `cron/deliveries.db` carries any tested
value), the shared-surface DB, the seven uncompressed rotation snapshots under `~/.hermes/mnemosyne/backups/`
(all seven reproduce the DB in full) and a chunked scan of the **1.7 GB session store `state.db`**, which
carries all four values and is the likely capture origin (capture writes verbatim session text). **Nothing was
rotated, deleted or scrubbed; no credential value was printed, logged or written to any page.**
- **Both 11-04 casualties recovered in full.** `85edf6772bca` (*Tech & AI Newsletter Digest*) completed its 07:00
run in **7 m 01 s** (07:00:42 -> 07:07:43) and `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) in
**5 h 35 m 35 s** (07:00:42 -> 12:36:18); neither re-raised `RuntimeError: Upstream stream failed before
completion.` The 11-05 ledger settled **175 rows / 18 jobs - 173 `completed` / 2 `running` / 0 `failed`**, the
**first zero-`failed` day since 2026-11-02**; the two `running` rows are `ea79a2107e36` (*Mnemosyne Health
Monitor*, 12:00:45 slot, still in flight 12 h+ with no delivery) and this audit (`cbfe434a3b5e`). Deliveries
**10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`,
*Uber Eats Monthly Spend Report*, `delivery_failed`, streak 0).
- **Incident ledger emptied.** Unclosed rows **3 -> 0**: the digest row closed **07:07:42** and both Job Radar rows
(`dca848_dfd4bea84f2a` `timeout`, `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**.
`incidents/2026-11-03-job-radar-stream-active-timeout.md` -> `status: resolved` with a Resolution section;
`incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` closed out - the first empty ledger since
2026-11-02.
- **Updated [[systems/scheduled-tasks]]** - frontmatter (`updated` / `verified_on` -> 2026-11-06, `review_after`
2026-11-07, `live-sessions-db` added to `sources`), a new `## Count` bullet for the 2026-11-06 read and a new
`## Re-verified 2026-11-06` span (recoveries, zero-`failed` ledger, delivery ledger, the secrets finding).
- **Updated [[systems/mnemosyne]]** - frontmatter dates -> 2026-11-06 (`review_after` 2026-11-13), `sources` +
`live-secrets-scan, live-state-db`, and a **security entry at the head of `## Known Issues`** recording that the
store's own tables carry 17 rows of live credential material, that recall is the exposure path, and that only
rotation covers the copies already taken.
- **Updated [[current-state]], [[index]]** (`## Incidents` link for the new page, header summary rewritten to the
2026-11-06 read) and this log.
- **Freshness:** `raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes) -> **0 stale (>=90d),
124 fresh, 3 no-date of 127 pages**, the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
`experiments/active/example.md`, `log.md`).
- **Not done / unchanged:** no page removed or archived; no decision reversed; `066bb5191280` and the
`deepseek-paid-smart`-on-`omniroute` dispatch fault carried, not re-diagnosed. **No secrets, passwords, tokens
or keys written or quoted** - the incident page records where secrets live, never their values.
## 2026-11-06 03:0x AWST - stocktake (wiki-stocktake-maintenance)
- **Ran `wiki-maintain.sh` (repo script, canonical path).** `systems/skills-index.md` regenerated ->
**302 skills / 68 categories** (unchanged count; date rolled to 2026-11-06). Freshness audit at the
**21-day** threshold -> **0 stale / 124 fresh / 3 no-date of 127 pages**. Quartz rebuilt (761 files emitted,
127 processed) and deployed to `public/`; `hermes-wiki-serve.service` restarted. Script exit 0.
⚠ **Threshold correction:** the previously committed `freshness-20261106.txt` carried a **90-day** threshold
(the nightly audit's setting); the stocktake script's threshold is **21 days**, and this run's report is the
authoritative one for today.
- **Freshness: 0 stale pages.** The three no-date pages are unchanged (`SCHEMA.md`,
`experiments/active/example.md`, `log.md`). No page is over 21 days.
- **Live-vs-page drift diff (the age scan alone would have missed all of this).**
- **[[infrastructure/tailscale]] - 1 state transition + 8 offline-age drifts + 2 dead Serve routes.**
`anthonys-mac-mini` **came back online** (peer record `Online: true`; `tailscale ping 100.76.60.37` ->
`pong ... via DERP(syd) in 103ms`). Offline ages refreshed +1d each: cloudreve 103->**104d**,
elementsynapse 106->**107d**, freshrss-new 79->**80d**, octoagent 3->**4d**, octop 1->**2d**,
omarchybook 14->**15d**, omarchymac 3->**4d**, openmausbot 14->**15d**. **Every peer IP still matches the
live JSON exactly; no live peer missing; no orphan row.** `m93p` re-pinged -> pong (3ms); `octop`
re-pinged -> timeout, matching its `LastSeen`. Container states cross-checked against live `pct list` on
**both** hypervisors (pve 17 guests, m93p 19 guests + 2 VMs). ⚠ **`/agent-estate` -> 8769 and
`/synergy-prep` -> 9123 are DEAD** (no listener) while still present in `tailscale serve status` - both
routes now marked DEAD on the page, matching [[infrastructure/network-map]] which had already recorded
them as dead on 2026-11-05. Every other Serve target has a live listener.
- **[[infrastructure/docker-services]] - the 2026-11-05 "the seeder does not exist on any host" finding was
FALSE; corrected on the page with strikethrough + dated correction.** The `worldmonitor-seeder-*`
containers are **transient and systemd-timer driven** on CT200 (`worldmonitor-seed.timer`,
`OnUnitInactiveSec=4h` -> `worldmonitor-seeder-scheduled`; `worldmonitor-seed-fast.timer`,
`OnUnitInactiveSec=15min` -> `worldmonitor-seeder-fast`), so they are visible only while a run is in
flight. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x **neither** was - hence
the false negative. The "no worldmonitor-seeder cron job" claim was also wrong: the driver is a **systemd
timer pair, not cron**. The earlier "rename" narrative was wrong too: **one script, two timer-driven
container names**. CT200's 61-container running set still matches live **set-wise** (no drift). **CT333's
container renamed again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z).
CT201 20/26, CT460 1/1, CT1001 1/9, CT333 8/9 - all unchanged and correct.
- **[[systems/scheduled-tasks]] - no drift.** Live `jobs.json` reads **29 jobs / 26 enabled / 3 disabled**,
**1** non-clean `last_status` (`066bb5191280` Uber Eats Monthly Spend Report, `delivery_failed`, streak 0) -
exactly what the page already records.
- **[[infrastructure/hosts]] - cron count 29/26/3 still correct** (last re-verified 2026-10-31).
- **Updated [[current-state]] and [[index]]** - the false seeder claim and the "every state matching" Tailscale
claim are now marked superseded with dated corrections rather than silently rewritten (SCHEMA rule 9).
- **Not done / unchanged:** no page removed or archived; no decision reversed; the open
`deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not
re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or
quoted.**
- **Follow-up (04:0x AWST): the one page sitting exactly on the 21-day threshold was re-verified rather
than left to expire.** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` was at **21d** and
would have tripped the threshold tomorrow. Its residual claim ("`omniroute-1` answers again") was
re-probed **live** and still holds: TCP connect to `100.78.63.20:20128` succeeds (no `Errno 111`),
the Tailscale peer reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**.
Page re-verified (`updated`/`verified_on` -> 2026-11-06, `review_after` -> 2026-12-06) with the probe
recorded. ⚠ **New trap recorded on the page and in [[systems/scheduled-tasks]]:** on `omniroute-1` the
**raw** model id `deepseek/deepseek-v4-flash` returns **HTTP 401 `No active credentials for provider:
deepseek`**, while the fleet's **`deepseek-paid-smart` alias returns HTTP 200** and resolves to that same
model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the
raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for
`omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.)
## [2026-11-07] audit - window 2026-11-06
**Headline: a red row with no alert.** 2026-11-06 settled **172 execution rows across 16 distinct jobs - 170
`completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart** signature:
the Job Radar's **11:30 slot** (`5fc292ff0f3b`, row `95347fa5…`, 11:30:32 -> 13:26:44) was **discarded as
stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job
(08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. Every previous
occurrence of this string came from a restart/reload orphaning a live run; **this one had no restart** - the
claim was lost to another fire of the same job. The failed row raised **no `cron_incidents` row** (newest
ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted**. New page ->
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]].
- **Ledger/day:** deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3
disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), the
**fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, fleet `updated_at`
**2026-11-07T00:39:46+08:00**. Non-clean `last_status` unchanged at **1** (`066bb5191280`,
`delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0**; the one open page remains
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window** - no
rotation, deletion or scrub).
- **Job Radar recovered the day in full:** the 14:30 slot ran a normal **12 m 27 s** cycle and delivered at
14:42:59 - `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`,
cache total **494**, 10 today-dated rows, 20 non-empty artifacts, *Run status: COMPLETE*.
- **The stocktake's overrun streak fully unwound:** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m
03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Stage 1 (`dca8482e4f76`) ran
clean for a **second consecutive day** (07:00:44 -> 08:29:21, 1 h 28 m 37 s, `suppressed`), so the 11-04
Open entry about its two lost candidate sets is **cleared in substance** - the 11-03 ledger row closed
`resolved` at 2026-11-05T12:36:17 and both runs since have been clean.
- **The last audit's open question answered itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after 12 h
13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. Ballooned on
consecutive days, never lost, no incident row - **watching, not open.**
- **Sessions:** **15 cron sessions started in-window (495 messages / 273 tool calls)**, 16 sessions with
activity (incl. one `desktop` session, 146 messages). Busiest: `wiki-stocktake-maintenance` 157, Job Radar
Discovery 92, `Enhanced Weekend Adventure Planner` 82, Job Radar Write and Letters (14:30) 28, Mnemosyne
Health Monitor 27.
- **Wiki maintenance:** `wiki-maintain.sh` run **after** this audit's writes (freshness report
`raw/stocktake/freshness-20261107.txt`); skills-index regenerated; Quartz rebuilt. Pages touched:
[[systems/scheduled-tasks]] (+frontmatter, +Count bullet, +`Re-verified 2026-11-07` section),
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
mirror `/home/hermes/wiki` updated (both conventions).
## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted.
- **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold.
(`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims.
- **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**.
- **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`).
Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**.
Both hosts run the identical **`pve-manager 9.2.10`** build.
- **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal).
The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**).
- **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**.
- **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running.
- **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
and **21** on `custom:omni-101`.
- **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`.
A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**;
`config.yaml` still defaults to `custom:omni-101`.
- **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
found; every drift above was a documentation drift corrected against a live probe.
## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
- **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`.
- **Four independent root causes, all confirmed against live systems:**
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`.
2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct;
**the script was the liar**. This is why the outage ran 92 days without a job-level red.
3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
`.gitignore` already excluded for *new* commits but that history still carried.
4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists.
- **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip.
Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
`origin/main` and never resent. Pre-rewrite history preserved under tag
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
every unpushed blob: clean.**
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
and passed via the environment, never argv).
- **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`;
`rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony).
Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
→ pushed + verified; already-up-to-date → no push, exit 0.
- **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
[[systems/scheduled-tasks]] (job row → RESOLVED), this log.
- **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
included in this commit so the tree is not left dirty.
## [2026-11-08] audit - window 2026-11-07
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
0 `for_failure`.
- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13).
- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52.
- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`);
the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
`pid 3109529`). CT450's install stamp now also reads **0.23.0**.
- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
an auth error.
- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed.
- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy).
- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`.
- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
(`/home/hermes/wiki`).
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
**2026-11-08 00:38:29 AWST**.
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script) run **after** this audit's writes —
`systems/skills-index.md` regenerated (**299 skills / 67 categories**; disk shows 300 `SKILL.md`, one nested
duplicate de-duped by the generator), freshness report `raw/stocktake/freshness-20261109.txt`
(**0 stale / 127 fresh / 3 no-date of 130 pages**), Quartz rebuilt (**130 markdown → 793 files** deployed to
`public/`) and `hermes-wiki-serve.service` (a **user** unit — `systemctl --user`) restarted
**2026-11-09 00:18:30 AWST**. Published pages verified **HTTP 200** at `http://127.0.0.1:9120/<path>` **and**
on the external route `https://hermes.kangaroo-eel.ts.net/wiki/<path>`.
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
the upstream error code only.
## [2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day
- **New page:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` — **RESOLVED same day.**
- **The finding:** `conduit_push` is the **only** `pre_tool_call` registrant on the host
(`plugins/conduit_push/__init__.py:119`), and its callback overran Hermes's built-in **30 s** hook budget
**58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `~/.hermes/logs/errors.log`
holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while
still running"* suppression lines. Every tool call degraded from slow to suppressed.
- **The fix (2026-11-08 13:35 AWST):** `plugins.hook_callback_timeout` **30 → 120** in
`~/.hermes/config.yaml`. **Verified live**, and **verified effective** — the last timeout line is still
08:21:03, re-checked at audit time.
- **A plausible diagnosis measured and rejected:** the plugin's caching was *not* the fault. Timed
in-process: **3.1 s cold, 0.00 s warm** (`session_titles.py:21 CACHE_SECONDS = 30.0`). The
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the
2-core CT460 (`load average 5.89` at 13:27:50; `state.db` checkpointing a **40 MB WAL** on the same
spindle). **Next lever is `state.db` checkpoint/retention discipline, not more caching.**
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and
**12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday
Briefing* `ecc1fc13655a` took **2 h 09 m 53 s**; and the stocktake failed a third day.
- **Cron ledger:** 11-08 settled **169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit,
23:59:33). The single failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08**, on
a **new terminal signature**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still
pending and the model produced no follow-up text.` (the previous two red days read *"lost its durable fire
claim ownership"*). **`failure_streak: 3`.** *Working hypothesis, not proven:* a suppressed `pre_tool_call`
holding a tool result open is exactly what the error describes, but the string does not name the hook.
Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**. Fleet **29 jobs (26 enabled / 3 disabled)** — a
**sixth** consecutive audit at the same count, nothing added, retired, disabled or repinned (fleet
`updated_at` moved to `2026-11-09T00:06:50+08:00`, read as a run-state stamp only).
- **Incidents:** unclosed rows **2 → 3** — a third silent `detected` row for the stocktake
(`1a8e90_a9cad99cb150`, `alerted_at: null`), all three still unreconciled. **The alerting path is not at
fault:** the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and closed `resolved` at
**08:06:54** (its admission-read `database is locked`, cleared when the run completed).
- ✅ **Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works.** The 18:00 slot
(agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against
**105,982 B** for the last agent-mode run. Coverage live at **working 1030/1030 = 100.0 %**, **episodic
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. The script compares with `awk` because
**`bc` is not installed on this host**.
- ✅ **Verified resolution 2 — `state.db` re-measured and one corruption remnant removed.** Now
**1,890,656,256 B (1.89 GB)**, **2,901 sessions / 197,682 messages**, live **40,157,672 B** WAL.
`state.db.corrupted.20260811_151052` (**624.1 MB** + 32 KB `-shm` + 0 B `-wal`) **deleted** in-window;
`state.db.recovered.recovery.json` (14,566 B) remains and
`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (**23.6 MB**) was **not** cleaned.
- **Pages updated:** `systems/conduit.md` (the pre-tool-call section, **total**), `systems/hermes-agent.md`
(hook budget + `state.db` size, **total**), `systems/scheduled-tasks.md` (new Count bullet, **total**),
`systems/mnemosyne.md` (health-monitor conversion verified, **total**), `current-state.md` (**total**),
`incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md` (`state.db` 1.7 → 1.89 GB in three
places, re `confident`), `incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md` (open items),
`index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log.
- **No wiki page removed or archived.** No system obsoleted; no decision reversed.
- **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit.
### 2026-11-10 — Nightly audit (run 2026-11-09 23:59 AWST; window: 2026-11-09, live probes 00:0x–01:0x)
- **2026-11-09 is the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Deliveries **8/8 `delivered`**, 0 `for_failure`. Unclosed incident rows **3 → 0** (`cron_incidents`: 134 rows, 0 open). → `current-state.md`, [[systems/scheduled-tasks]]
- ⚠️ **The stocktake's `failure_streak` 3 → 0 / `last_status ok` are cosmetic.** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m**, closed `completed` (`suppressed`), and landed **zero edits and no commit** — its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09); it is still pinned to `custom:omni-101` / `free-stack`. New page: [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. The three long-open incident rows (`1a8e90_628d914f1f97`, `_2f370ef764f5`, `_a9cad99cb150`) all closed at **12:10:47.125205**, the instant the run ended — hot-loop alert closing, not a fix.
- ⚠️ **New finding — Job Radar Stage 1 (11-09)** `dca8482e4f76` ran **07:00:48 → 16:10:12 = 9 h 09 m**, never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**), and both Stage-2 slots (`5fc292ff0f3b`) fired **before** it finished and declined silently. Same silent family as 11-02/11-03 with a new trigger (read loop). No incident row raised. New page: [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
- ✅ **11-08 `pre_tool_call` hook cascade did not recur** — no `pre_tool_call` timeout line in `errors.log`; only two benign skips; no 11-09 run outside the two long-runners exceeded 74 min. Follow-up appended to [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
- **Tailscale:** 58 non-funnel nodes (57 peers + `hermes`), 42 online. `octop` **online** again; 8 offline ages refreshed (cloudreve 108d, elementsynapse 111d, freshrss-new 84d, conduit 65d, octoagent 8d, omarchybook 19d, openmausbot 19d, ubuntu-server-xfce 81d); `codex-environment-*` last-seen stamps corrected. → [[infrastructure/tailscale]]
- **OmniRoute:** `omniroute-1` CT101 `/v1/combos` → HTTP 200, **55 combos**, `free-stack` present. `/v1/models` on both nodes and CT600 `/v1/combos` → **HTTP 401**, so the **6,480-model catalog figure (2026-10-26) stays unverified this window; no page changed.** → [[systems/omniroute]]
- **Not verified (gap recorded):** CT450/CT460 Hermes version strings (the `pct exec` probe resolves the stale root-level `/usr/local/bin/hermes`; CT460's `hermes` isn't on PATH in that shell) and their docker `Up` days (`lxc-attach … Failed to exec "docker"`) — [[infrastructure/hosts]] and [[infrastructure/docker-services]] version/uptime lines stand as written. Unchanged.
- No secrets, keys, tokens or passwords written.
- **Post-audit housekeeping (01:0x AWST).** Vault sync committed in `/home/hermes/wiki` (**`3a19147`**, both vault conventions; 20 files, then amended with the freshness report). `wiki-maintain.sh` re-run **after** the page-date corrections → **skills 299 / 67 categories** (`systems/skills-index.md`), freshness **2026-11-10: 0 stale / 127 fresh / 3 no-date of 132**, Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. Two long-stale pages re-verified and de-flagged: [[systems/push-notifications]] (bridge sha256 `59cabfcc…` unchanged; `vory-push` relay **live 01:00:34, embedded in another Hermes process** — a `pgrep` on the script name is *expected* to return nothing while healthy) and [[incidents/2026-10-18-degenerate-stream-canned-substitute]] (6 runs since 10-18, **0 failed**; newest 2026-11-09 12:00:41 `completed`). **Pushed:** `6de35a3` (maintenance) → this.
### 2026-11-11 — Stocktake: 36 stale pages re-verified; 0 stale; one materially stale fallback chain rewritten
- **Freshness:** the 03:00 `wiki-maintain.sh` run flagged **36 stale / 93 fresh / 3 no-date of 132** — the whole 2026-10-20 cohort (runbooks, concepts, decisions, comparisons, experiments, seven systems pages, the Gmail + Mnemosyne-Capture incidents, `CONTRIBUTING.md`, `agents/index.md`). After the corrections below, the **same-day report was regenerated** (script → fix → re-run) and reads **0 stale / 129 fresh / 3 no-date of 132**. Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted.
- **Skills:** `systems/skills-index.md` regenerates at **299 skills / 67 categories** — unchanged count from 2026-11-10; the only diff is the date bump. Verified the generator's arithmetic against the live tree by hand: **300 `SKILL.md` files** (excluding `.archive/`), **1 nested duplicate** (`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` duplicating the same category+name) → **299 unique (category, skill) pairs**. The count is correct; do not "fix" it to 300 or 303.
- ⚠️ **Materially stale page rewritten — [[runbooks/test-openrouter]].** The 2026-10-20 fallback chain was wrong in four of six slots. Live `hermes fallback list` (verbatim recorded on the page): primary **`smart-route` via `custom:omni-101`**, then `meituan/longcat-2.5-preview:free` (nous), `deepseek-v4.1-flash` (opencode-zen), `deepseek-v4.1-flash` (custom:guacano), `kilo-reflection-beam-free` (custom:litellm), `poolside/laguna-m.1:free` (kilocode), `or-inclusionai_ling-2_6-1t` (custom). `mimo-v2.5`, `stepfun/step-3.7-flash:free` and `qwen3.5:397b` via ollama-cloud have **left** the chain; two new entries **joined** at 3 and 4. `ollama.com` dropped out of the runbook's URL test list; `guacano`/`litellm` are noted as **internal + key-gated (401 without a key)**. → [[concepts/model-routing]], [[comparisons/model-routers]]
- ⚠️ **A false claim corrected — [[comparisons/hermes-vs-nanobot]].** The 2026-10-20 note claimed "CT333 is `zeroclaw` in the current Proxmox naming". Live `pct list` **and** `pct exec 333 -- hostname` both return **`clawtest`** — the wiki's original label was right; `zeroclaw` is only the daemon running inside it. The page now says so, dated.
- ⚠️ **Incident signature change — [[incidents/2026-10-22-nous-portal-refresh-token-invalid]].** The refresh-token failure is **gone** (zero `invalid_grant` lines in `errors.log` through 2026-11-10; `auth.json` live, no `NOUS_API_KEY` in `.env`). What replaced it: **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5`, 3× on 2026-11-10. Cause identified: a **ninth auxiliary slot** `auxiliary.pulse_memory` points back at **`nous`** with a paid model — the 2026-10-25 remediation moved the original eight slots to `custom:omni-101` but this slot was added afterwards and is the only active nous path left. `nous` is still first in `fallback_providers`, but on a *free* model not observed failing. Open item revised from "re-authenticate" to "move `pulse_memory` or add credits".
- **[[incidents/2026-08-29-gmail-read-state]] — the cron error state cleared, the root cause did not.** `goimapnotify` is **still active** (now PID 441), the Important-mail monitor (`d40085da631c`) is **still paused**, and `fetch_newsletters.py` **still calls `himalaya message read --preview`** on Anthony's INBOX — 74 days unfixed. The three email crons that read `error` on 2026-10-20 (`85edf6772bca`, `08eb86f02d37`, `06a0a1c8704f`) are now `ok` with `failure_streak: 0`, but green cron status is not evidence of safe behaviour: a job that marks mail read *and succeeds* still pollutes the mailbox. Both original mitigations are now inert.
- **Live-vs-page diffs run (per skill pattern 12a), all claims held:** ct207 `cloudflared` ingress re-read in full (four Hermes entries unchanged, `mcp-portal` still absent — [[systems/cloudflare-tunnel]], [[infrastructure/domains-and-tunnels]]); CT205 Guanaco at `v0.8.15`/`5df435e` with Ollama 200 ([[systems/guanaco]]); CT270 `occ status` 33.0.2.2 installed/not-in-maintenance ([[systems/nextcloud]]); headless stack process/port state ([[systems/headless-server]], [[runbooks/restart-browser]]); hermes-ui 5174 on both CT460 and CT450 ([[systems/hermes-ui]]); all five `hermes-*` user units active + gateway/WebUI/Camofox effective cgroup limits ([[runbooks/restart-hermes]], [[runbooks/gateway-resource-resilience]]); `tool_loop_guardrails` and the whole `terminal:`/`tools:`/`memory.mnemosyne`/`custom_providers` blocks byte-identical; `ops` profile **still deleted 2026-10-11** while `multiplex_profiles: true` and both `profile_routes` still point at it — **31 days unresolved** ([[runbooks/multiplexer-setup]]); `/home/hermes/wiki` still a local-only repo, `/home/hermes/wiki-backups/` still absent, `~/.hermes/backups/` incident dirs intact; `m93p` **not** re-probed this pass (its 2026-10-20 guest list stands as written).
- **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`.
- **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]).
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists").
## 2026-11-12 00:1x AWST - Nightly wiki audit (window 2026-11-10 23:59 -> 2026-11-11 23:59)
- **The audit's own failure is the headline.** The 2026-11-10 23:59 slot (`cbfe434a3b5e`) ran **12 m 35 s** (23:59:54.486572 -> 2026-11-11 00:12:29.646987) and died on the router's pre-dispatch filter: `HTTP 429 ... all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded, delivery `suppressed`. **Consequence: the 2026-11-10 window was never reviewed.** Recovery was implicit (pool reset ≈ **06:36:57**; the 07:0x cohort ran clean), and this audit deliberately covers **both** windows. New page: [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]; new index entry.
- **The raised row was never alerted.** `cbfe43_ed01845310d8` (`rate_limit`) sits at `detected` with **`alerted_at: NULL`, `closed_at: NULL`** — the **only open row** in `cron_incidents` (the previous one, `5fc292_773822455856`, closed `resolved` at 08:03:47). Fleet non-clean `last_status` **1 -> 2** (`066bb5191280` Uber Eats `delivery_failed` streak 0, plus `cbfe434a3b5e` `error` streak 1).
- **Both long-running incidents recovered, with output.** Stocktake `1a8e90ea2271` **03:00:56.081504 -> 04:11:38.724852 = 1 h 10 m 42 s**, `completed`, **36 stale pages re-verified -> 0 stale**, commits **`341dda9`** + **`c108eb1`** (freshness report `raw/stocktake/freshness-20261111.txt`, 6,235 B) — closing [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. Job Radar Stage 1 `dca8482e4f76` **07:01:08.358004 -> 07:37:34.076224 = 36 m 26 s**, writing the **first handoff since 2026-11-06** (`radar-ref/handoff/2026-11-11.json`, 22,682 B, mtime 07:59); Stage 2 `5fc292ff0f3b` recovery fire 07:35:56.632032 -> 08:03:47.816811 `delivered`, then 11:30:56 -> 11:35:18 and 14:30:55 -> 14:34:56 `delivered`, auto-closing the 11-10 row `resolved` — closing [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. Both incident pages set to `status: resolved` with a dated Recovery section.
- **Ledger, both days re-parsed (times stored as ISO TEXT, grouped by `substr(started_at,1,10)`).** 2026-11-09 **174 completed / 0 failed**; **2026-11-10: 157 rows — 155 `completed` / 2 `failed`** (Radar S2 scheduled 11:30:35.267155 -> 22:18:49.814215 = **10 h 48 m 14 s** `interrupted` with `for_failure: 1`; the audit's own 12 m 35 s 429); **2026-11-11: 172 rows — 171 `completed` / 1 `running` / 0 `failed`**, 15 distinct jobs, **first zero-failure day since 11-09**. Longest 11-11 run **1 h 10 m 42 s** — **nothing over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). Fleet **29 (26 enabled / 3 disabled)** for a sixth window.
- **Deliveries checked at the destination:** 2026-11-11 = **7 rows, 7 / 7 `delivered`, 0 `for_failure`** (07:11:42, 07:31:01, 08:03:47, 08:34:36, 12:19:38, 14:34:55, 18:45:58); 2026-11-10's late row (22:18:48) is the one `for_failure: 1`. 2026-11-12 so far: 5 rows, 4 `completed` / 1 `running`.
- **Providers unchanged; symptoms worse.** `config.yaml` mtime still **2026-11-09 22:58:07** — nothing repaired this window. The expired `opencode-zen` key at `fallback_providers[1]` produced **7 x HTTP 401 `api_key_expired`** on 11-11 (07:01:13 -> 19:19:39), each **immediately after** nous returned **14 x 404 `insufficient_credits_for_paid_model`** (07:01:11 -> last 19:19:38, all from the still-open desktop session `20261031_190813_95fbde`). **Two consecutive dead hops on one chain** — [[systems/opencode-go]] updated from "latent risk" to observed failure; [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] gained the 11-11 escalation block. `auxiliary.pulse_memory` remains the **only slot with no `fallback_chain`** and logged **19 lines / 11 refusals** on 11-11.
- **Pages written:** 1 new incident; 2 incidents -> `resolved`; 2 system pages + `current-state.md` + `index.md` updated; `systems/scheduled-tasks.md` gained the 2026-11-11 count/state normalisation. `~/.hermes/logs/errors.log` is now **6,179 lines** through **2026-11-12 00:10:11**. **No page deleted, no decision reversed, no secrets written** — no credential value appears in any page; the opencode key is recorded by *state*, never by value.
- **Post-audit housekeeping (00:2x AWST 2026-11-12), vault sync.** Touched files copied to **both** vault conventions — `…/agent-estate-wiki/<rel>` and `…/agent-estate-wiki/wiki/<rel>` — with `sha256sum` confirmed equal across all three copies; vault committed locally in `/home/hermes/wiki` (`master`, **no remote — never pushed**), explicit paths only, so the unrelated `reports/uber-eats/*` dirt stayed out. Before copying, each vault file was compared against the **pre-edit** authoring blob (`c108eb1`): every difference was a *stale-but-clean* older version (vault `index.md`/`skills-index.md` still 11-10/11-09, `log.md` still missing the 11-11 stocktake entry, the nous incident page still 2026-10-26 at 48 lines vs 96) — **no direct-to-vault edits to preserve**, so forward-sync was safe. The 2026-10-22 nous incident page was **created** in vault B's `incidents/` (it had never been mirrored); the new 11-10 audit page went into **both** `incidents/` dirs.
- **Vault drift re-measured (hash sweep over all 133 authoring pages, this run's 10 touched files excluded):** **Vault A — 13 stale / 56 missing / 64 in sync. Vault B — 58 stale / 8 missing / 67 in sync.** Unchanged in character from the 2026-10-19 measurement (the copy bridge still only syncs what an audit touches), so the missing-page backlog is recorded here and **left to an explicit request** rather than silently swept — a 133-page mirror pass is bigger than a nightly audit's mandate. ⚠ This is now a two-night-old gap: because the 2026-11-10 audit died at 00:12, **nothing was mirrored for that window either.**
- **Freshness report:** `raw/stocktake/freshness-20261112.txt` (this run, 6,325 B) — **1 stale / 129 fresh / 3 no-date of 133** pages (was 132; **+1 = the new incident page**). The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding. `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. Quartz rebuilt and `hermes-wiki-serve.service` restarted; the new page returns **200** at the served site with its body content present.
## 2026-11-12 03:0x–04:1x AWST — Stocktake (03:00): tailscale drift corrected; 1 stale + 5 threshold pages re-verified live
- **Freshness:** `wiki-maintain.sh` → **1 stale / 129 fresh / 3 no-date of 133** (was 132, +1 = the new 2026-11-10 incident page). The single stale page was **[[incidents/2026-09-02-sftpgo-outage]] at 22d** — re-verified live and cleared. The 3 no-date pages are unchanged (`SCHEMA.md`, `experiments/active/example.md`, `log.md`). `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**.
- **⚠ Live-vs-page diff found real drift in [[infrastructure/tailscale.md]] (2d old — age scan would have missed it):** three peers read `Online: false` in the peer record while **all three ping fine** — `mailflow` (CT1262 m93p, pong `192.168.178.142:41641` 2 ms), `qwenpaw` (CT465 pve, pong `192.168.178.186:41641` 7 ms) and `springroll` (pong via **DERP(sin)** 210 ms, relay not direct) — with both LXCs `running` in `pct list`. All three rows stay **online**; the peer record is stale (the m93p pattern again). One **genuine** correction in the other direction: `omarchymac`'s row claimed `offline 2d / last seen 2026-11-08 00:45Z` when the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** and ping → **timeout**. 9 named offline ages + the six `codex-environment-*` ephemerals refreshed (+2d). No IP changes, no missing rows, no new peers (58 rows = 57 live IPv4 peers + `hermes`; 80 peers total / 57 IPv4 / 23 IPv6-only, unchanged). Table rebuilt by script, not by hand.
- **Re-probed and re-confirmed:** CT299 `sftpgo` **running**, peer `Online: true`, `pong via 192.168.178.92:41641 in 0s`, healthz **401** (auth-gated, service up) — closing [[incidents/2026-09-02-sftpgo-outage]] as still-resolved, and noting that `pct config 299` now reads **`swap: 512`** (the 43 Gi over-provisioning in that root cause is gone). Both DEAD Serve targets (`8769` `/agent-estate`, `9123` `/synergy-prep`) still have **no listener** — re-confirmed a fifth time. CT201's Serve table re-probed unchanged (`/`→8085, `/mgr`→8084 on both `:443` and `:8443`). Container counts re-confirmed on **both** hypervisors incl. `qm list`: `pve` **17 LXCs + 4 VMs**, `m93p` **20 LXCs + 2 VMs**. **New guest found: CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it runs without joining the tailnet (same class as CT105 conduit, CT1264 openmausbot).
- **Incident page re-opened for a **recurrence** — [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] (21d, tomorrow's stale list).** The *"all targets were skipped by pre-dispatch filters"* signature has now fired on a **second job and a second pin**: `cbfe434a3b5e` (Wiki Audit) failed on **2026-11-10** with `RuntimeError: HTTP 429: … pre-dispatch filters … (credits 100%)`, `failure_type: rate_limit`, and its incident row `cbfe43_ed01845310d8` closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean on 11-06/09/10/11. Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is *not* re-readable from the DB — the new read is in-window only.
- **Four threshold/near-threshold pages re-verified against live state (skill pattern 22), each with a dated note — no date bump without a probe:**
- [[incidents/2026-07-29-mnemosyne-relay]] (19d) — `tools: None` absent from `config.yaml`; **0** occurrences of `Unknown Mnemosyne tool` across 4 rotated gateway logs and **0** of the relay `scope handle` error; `mnemosyne stats` → 2782 memories and recall returns ranked results.
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] (18d) — `himalaya envelope list -a work -s 1` still fails on `login not supported`; no OAuth2 token in the himalaya config, so path 1 (Anthony's sign-in) remains the only route.
- [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] (19d) — keyed probes on `omniroute-101`: **`good-free` → HTTP 200 (`glm-5.2-cheap:free`)** and **`free-stack` → HTTP 200 (`stepfun-ai/step-5-preview`)**. The mitigation holds and neither combo now resolves to deepseek. **The three free-DeepSeek upstreams themselves were NOT probed**, so "upstream still degraded" is neither confirmed nor refuted and restoring deepseek-first ordering is still unjustified — recorded as such on the page.
- [[reports/communities-wa-social-media-audit-2026]] (18d) — **not re-verified this pass** (live Department of Communities pages not re-fetched); its `review_after: 2026-11-26` still stands and its own note already scopes the unverified claims. Flagged, left as written.
- **Order of operations:** script → page corrections → **re-run script** → log/index → commit. `wiki-maintain.sh` re-run after every page edit; Quartz rebuilt and `hermes-wiki-serve.service` restarted.
- **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page; the OmniRoute probe used the `key_env` name, never the value.
- **Correction to the entry above (2026-11-12 00:1x AWST audit), verified live rather than inherited:** that entry states *"The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding."* **That is wrong.** The committed `raw/stocktake/freshness-20261112.txt` from that run (sha256 `da5f9e2e…`, byte-identical to this run's pre-fix output) reads **`- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21)`** as its single `## STALE (1)` line — the real page, which this stocktake has now re-verified and cleared. `log.md` sits in the `## NO updated DATE` bucket, as always (pattern 13). The stale page was therefore **left unflagged in prose for one cycle while sitting stale in the tree** — the exact "fresh-looking prose, stale content" failure the report exists to prevent.
## 2026-11-12 23:5x – 2026-11-13 00:4x AWST — Nightly audit (window 2026-11-11 23:59 → 2026-11-12 23:59)
**Clean day, incident-clean estate — and an upgrade the security scan would not allow.**
- **The estate's last open incident closed itself.** `cbfe43_ed01845310d8` (the 2026-11-10 audit's 429)
went `closed_at: 2026-11-12T00:25:36.133387+08:00`, `state: resolved`, when the **next slot of the same
job ran clean** — `cbfe434a3b5e` 2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = **25 m 42 s**,
`completed`, delivery carrying both windows. **`cron_incidents`: 0 unclosed of 137.** No config produced
it; the pool reset at ≈ 06:36:57 on 11-11 did. The 429 page's `status: open` was **stale** — corrected to
`resolved` with a new *Resolution* section, and its "only open row" wording superseded.
→ [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
- **Ledger, live from `executions.db`:** 2026-11-11 **172 / 172 `completed` / 0 `failed`**; 2026-11-12
**176 rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, 23:59:54); 2026-11-13 so far
7 / 7. **Second consecutive zero-`failed` day.** Deliveries **11-11: 7 / 7; 11-12: 9 / 9 `delivered`**,
0 `for_failure`. Fleet **29 (26 / 3)** — seventh window unchanged. Retention floor `2026-11-07T00:00:42`,
1,001 rows.
- **NEW PAGE — [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].** On 2026-11-12
(~17:38–18:29) `hermes plugins install … --enable --force` for `hermes-hq-push` was refused:
*"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec))"*, *"--force does not override a
dangerous verdict."* **Verified in source rather than assumed:** the block is one unconditional branch
(`tools/plugin_guard.py:406-418`); `--force` lifts `caution` only; **there is no per-verdict override,
env var, plugin trusted-source list or allowlist** — the only key is `plugins.scan_on_install` (default
`true`, currently unset), and setting it `false` disables scanning for *every* future install. The
`trust_level` machinery exists in `tools/skills_guard.py` (skills path) and is **not read** by
`plugin_guard.py:398`, which hardcodes `trust_level="community"`. Installed **v0.9.42 untouched,
`enabled`, still delivering**. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`,
`fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file) — pushed from a fork
because `gh` here is authenticated as `Tony0410` with **read-only** access upstream; local clone
`/home/hermes/workspace/hq-plugins-fix` (`90c9d28`).
- **`systems/hermes-hq-push.md` corrected against live probes:** version **0.9.27 → 0.9.42** in the
*Verified state* table (which still carried the 2026-11-04 reading), tree mtime **2026-11-07 06:42 →
2026-11-08 02:19** (measured), the config claim's **line 7556 → 13226** (block opens line 13210 — noted
as drifting), plus a new *2026-11-12 — refused upgrade* section and two new *Open* items.
- **The "stored hold-and-ask instruction" was never stored.** Recursive sweep of `~/.hermes`
(**184,716 files**; generated/venv/browser-profile/SQLite trees excluded) for `wait for my answer`,
`show me what it found`, `security scan flags`, `security-scan` → **zero matches in any editable text
file**; the sentence exists only in `state.db` **conversation history** (58 rows). **Nothing removed, no
policy reverted** — there was nothing persisted to revert. Walkthrough recorded as the new skill
`stored-instruction-search-and-removal`; installed skills **299 → 300** (`skills-index.md` regenerated
by `wiki-maintain.sh`).
- **Config: no drift.** Live `~/.hermes/config.yaml` is **byte-identical to the 2026-11-12 18:14:52
snapshot** (565,433 B). The only delta since the **11-09** snapshot is the **model catalogue block**
(lines 1916-1941): 6 entries removed, 9 added. Nothing unrelated moved.
- **Still open, unchanged and re-confirmed:** the *"all targets were skipped by pre-dispatch filters"*
signature recurred on a **second job and a second pin** (2026-11-12 03:00 stocktake), and every
model-calling job still rides `custom:omni-101` with no failover for the free-tier cohort
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
- **Pages touched:** `incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md` (new),
`incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md`, `systems/hermes-hq-push.md`,
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`, `systems/skills-index.md`
(regenerated). **No page deleted, no decision reversed, no secret written.**