248 KiB
Wiki Log
Chronological record of wiki actions. Append-only.
[2026-07-22] init | Fresh Hermes Systems Wiki
- Created technical wiki at
/home/hermes/wiki - Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
- Wrote SCHEMA.md with 11 operating rules
- Wrote index.md as curated Start here dashboard
- Reseeded system/infrastructure/runbook stubs for existing homelab
- Git initialized, local-only
- No secrets, passwords, tokens, or private keys written to vault
[2026-07-22] audit | External audit pass (Claude, at Anthony's request)
- Merged duplicate pages:
systems/omniRoute.md(empty stub) -> keptsystems/omniroute.md(populated); repointed 5 links - Merged duplicate pages:
runbooks/safely-update-hermes.md-> kept asrunbooks/update-hermes-safely.md(fuller content); repointed 2 links - Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
- Fixed a broken
review_after: 0001-01-01in decisions/keep-openrouter-as-fallback.md - New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
[2026-07-25] operations | Tool Search and gateway resilience
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
- Documented how new skill categories are derived from the first folder below
~/.hermes/skills/ - Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
- Recorded the I/O-controller delegation limitation so an ineffective
IOWeightis not mistaken for active protection - Added health checks, exact effective settings, backup location, and rollback procedure
- No secrets, tokens, or private credentials were written to the wiki
[2026-07-28] dashboard | Jobs Dashboard feature additions
- Fixed bulk status update (payload shape mismatch:
{ids, status}→{ids, updates: {status}}) - Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion
Notes / JD Summary - Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
- Updated Synergy and Dept of Communities to Phone Screen status
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
[2026-07-27] incident | Job Radar search and dashboard recovery
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require
MISSING_COVER_LETTER_COUNT=0 - Recorded the production cron model (
gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5) - Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both
:443Serve and:8443/webhookFunnel - Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
- Verified no secrets, API keys or private credentials were written to the wiki
[2026-07-29] update | Steel Browser deployment — browser backend switch
- New page:
systems/browser-backend.md— rewritten to reflect Steel Browser as primary, Camoufox as fallback - New page:
comparisons/steel-vs-camoufox.md— comparison of the two backends - Updated:
infrastructure/hosts.md— added CT201 and CT460 to verified hosts table - Updated:
infrastructure/docker-services.md— added Steel Browser compose file - Updated:
infrastructure/tailscale.md— added CT201 Steel serve routes - Updated:
index.md— browser-backend link description - Updated:
current-state.md— added 2026-07-29 changes section - Config change:
browser.cdp_urlset tohttp://100.96.244.39:9223in ops profile
[2026-07-30] update | Open WebUI updated on CT245
- Updated image to
ghcr.io/open-webui/open-webui:main(new digest e97bf9531916, Jul 27) - Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
- Restarted container after OOM hang
- Updated
infrastructure/proxmox.md
[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
- Ran
mnemosyne reindex --yesto rebuild all embeddings — working memory now at 100% - Updated health check script (
mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py) - Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
- DB grew from 8.7M to 21M with the new embeddings
[2026-08-01] incident | Mnemosyne capture pipeline repaired
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no
working_memoryrow existed. - Corrected stringified structured YAML (
sync_roles,skip_contexts,reflect,sleep_threshold,ignore_patterns) and enabled profile isolation;hermes config checkpassed. - Verified the repair with a second CLI marker that persisted as an exact normalized
[USER]database row. - Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route
:9119; added authoritative identity/preference/workflow slots. - Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
- Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.
[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
- Rewrote comparisons/steel-vs-camoufox — full Steel experiment story, timeline, lessons learned
- Rewrote systems/browser-backend — Camoufox primary, Steel disabled
- Updated current-state — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
- Updated concepts/browser-automation — added Steel experiment lesson
- Created decisions/rollback-steel-to-camofox — formal decision record
- Updated systems/scheduled-tasks — job count 25, Uber Eats upgrade, GH Trending Daily removed
- Updated systems/mnemosyne — vector embedding coverage gap documented
- Updated systems/model-providers — DeepSeek V4 Flash 0731 details
- Created reports/uber-eats/index — landing page for monthly spend reports
- Updated decisions/index — linked new decision page
- Updated index.md — browser description, reports section, last-updated date
- Created nightly wiki audit cron job (23:59 AWST)
[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
- Ops profile (
~/.hermes/profiles/ops/config.yaml): enabledmnemosyneplugin withprofile_isolation: true,allow_tool_override: false,shared_surface_read: false - Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
- Verified end-to-end: memory writes land correctly in both profiles independently
- Updated systems/mnemosyne — profile isolation section now documents ops config
- Updated current-state — added ops profile Mnemosyne entry
- Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
- Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision
[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on
deepseek-flash-freeandsmart-route/good-freeacross both instances - Reworked
deepseek-flash-freeandgood-freeon CT600: strategypriority→fill-first, dropped proven culprits (238d5978/deepseek-v4-flash+-0731,nvidiadeepseek), healthy free models promoted, deepseek demoted to last resort - Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
- Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
- Updated systems/omniroute — added Recent Changes (2026-08-04) section
- Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated
updateddate
[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
- OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern.
is_active=0does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill. - Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
- Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
- Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
- Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
- Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
- Updated index.md — last-updated date 2026-08-04 → 2026-08-07
[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
- Portainer password reset documented: CT245 admin user confirmed as
anthony(notadmin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state. - SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint,
//root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.
[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
- Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from
freeapi/minimax-m2.7→smart-route(IDs:c8d52904e1d6,1b23c042e17d,d40085da631c). Mosquito Tracker fromfreeapi/auto→smart-route(ID:1b31187397a5). 9 stale-named jobs (e.g.vertex/gemini-2.5-flash,deepseek-v4-flash-free) →smart-route. Daily Marketing Job Radar →gpt-5.6-terra. - Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted
hermes-ui.service. Production build serves/ui/path correctly with relative asset paths.hermes-desktop.martinwa.orgidentified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460. - Phase One estate review completed: Manual cron run (ID
a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled withlast_status=ok, CT333 Nanobot has livejobs.jsonwith 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes. - Updated current-state — added 2026-08-08 changes section
- Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues
[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
- Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
- Root cause identified: Disk at 83% (11GB free on 65GB),
state.dbat 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues. - Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
- Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
- OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
- No new pages created — all changes applied to existing pages
[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
- OmniRoute AutoRoute variants documented:
auto/best-free(Auto Cheap) andauto/coding-freeare virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed:auto/best-freerouted to Antigravity. Updated systems/omniroute. - OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains
memory.provider: mnemosyne. Added to comparisons/memory-backends. - FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default
freellmapi-auto. Added to comparisons/model-routers. - No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.
[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
- OmniRoute migrated from CT600 to CT601: Provider renamed
custom:omniroute→custom:omni; base URLhttp://omniroute:20128/v1→http://omni:20128/v1. Hostomni.kangaroo-eel.ts.net(100.93.204.4, active) replacesomniroute.kangaroo-eel.ts.net(100.88.81.19, offline). CT600 stopped, CT601 running. DB path/var/lib/omniroute/storage.sqlite→/opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22)custom:omnirouterefs repointed tocustom:omni. Pre-migration backup at~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list,grep, Tailscale DNS). - Updated systems/omniroute — Current Configuration rewritten for
omni/CT601, added Recent Changes (2026-08-13), bumpedupdated/verified_onto 2026-08-13. - Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped
updated. - Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
- Updated infrastructure/hosts — OmniRoute proxy entry →
omni; CT600 reference marked verified/retired. - Updated index — last-updated date 2026-08-10 → 2026-08-13.
- No new pages created; all changes applied to existing pages. No secrets written.
[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
- Skills index drift incident:
systems/skills-index.mdclaimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creatingregenerate-skills-index.shscript, and adding cron job1a8e90ea2271(daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift. - Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped
updated/verified_onto 2026-08-15. - Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
- Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
- Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
- Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
- No secrets written.
[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
- New page:
systems/nextcloud.md— documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages. - Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
- Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
- Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
- No secrets written.
[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
- OmniRoute upstream capacity pressure documented:
ServiceUnavailableError: Structurally heavy chat request capacity is busyreported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded. - Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
- Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config
http://omni:20128/v1is correct —/v1is required. - Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped
updated/verified_onto 2026-08-16. - Updated current-state — added Recent Changes (2026-08-16) section, bumped
updatedto 2026-08-16. - Updated index.md — bumped last-updated date to 2026-08-16.
- No new pages created. No secrets written.
[2026-08-18] incident | Mnemosyne capture pipeline broken (active)
- New page:
incidents/2026-08-18-mnemosyne-capture-pipeline— working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions inrun_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification. - Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped
updated/verified_onto 2026-08-18, added cross-reference to Related. - Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped
updatedto 2026-08-18. - Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
- Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from
activetoresolved(that earlier incident was fixed). - No secrets written.
[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
- Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
wiki-stocktake-maintenance(1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed fromcx/gpt-5.6-luna→go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.- CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
- Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
- Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped
updated/verified_onto 2026-08-23. - Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped
updated/verified_onto 2026-08-23. - Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
- Updated index.md — bumped last-updated date to 2026-08-23.
- No secrets written.
[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
- New page:
systems/ignis.md— documented Ignis browser-native Obsidian (Dockerignis-subpath:0.8.9, port 8080, Tailscale Serve athermes.kangaroo-eel.ts.net/ignis). Volume mount:/home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active. - Updated systems/model-providers — added
command-codeprovider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumpedupdated/verified_onto 2026-08-19. - Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider,
deepseek-flash-mixcombo, and Tavili search proxy status. Bumpedupdated/verified_onto 2026-08-19. - Updated infrastructure/docker-services — added Ignis compose file and service definition.
- Updated infrastructure/hosts — added Ignis to CT460 row.
- Updated systems/obsidian — added Ignis as browser interface, corrected vault path to
agent-estate-wiki/, added Quartz static site reference. - Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
- Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
- No secrets written.
[2026-08-26] update | Nightly wiki audit — Cron job reconciliation
- Updated systems/scheduled-tasks — reconciled live
hermes cron list(27 jobs) against wiki table (29 entries). Removed:Workday start reminder(no longer in live list),CT 221 Miner Watchdog(jobd5516d0a5147removed; CT221 status unknown),Review Agent Estate Phase One(one-time job completed 2026-08-08). Added:weekly-save-restricted-bot-restart(0 3 * * 0, local, no-agent). Updated count from 29 to 27. - Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
- Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
- Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
- No secrets written.
[2026-09-02] incident | SFTPGo outage on CT299
- New page:
incidents/2026-09-02-sftpgo-outage.md— documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%),tailscaleddied; PVEpct startfailed due to 5 stalelxc-attachprocesses pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148,pct stop 299 --skiplock+pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP bannerSSH-2.0-SFTPGo_2.7.0. - Updated current-state — added Recent Changes (2026-09-02) section, bumped
updated/verified_onto 2026-09-04. - Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
- No secrets written.
[2026-08-28] update | Department of Communities role and social-media audit
- New page:
reports/communities-wa-social-media-audit-2026— durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role. - Updated systems/current-profile — recorded current employment context and linked the report.
- Updated index.md — added the report to the Reports section and refreshed the last-updated date.
- No secrets written.
[2026-09-20] update | Nightly wiki audit — No changes in past 24h
- No new pages created — no new systems, decisions, or incidents in the past 24 hours.
- No existing pages updated — all wiki content remains current.
- Freshness audit run:
freshness-20260920.txtgenerated at 03:01 AWST (scheduled cron). - Skills index auto-regenerated:
systems/skills-index.mdupdated at 03:01 AWST (scheduled cron1a8e90ea2271). - Scheduled tasks reconciled:
systems/scheduled-tasks.mdupdated at 03:02 AWST — 26 active jobs verified from livehermes cron list. - No secrets written.
[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
- New page
incidents/2026-09-28-hermes-managed-runtime-deps— the 2026-09-28 → 30 cron-fleet outage: jobs failed withRestart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused byModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in the uv-managed runtime venvs under~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2 (all 10 venvs,pip install --target), run every 15 min by job96eb95ce12f2. - Updated
systems/scheduled-tasks.md— table rebuilt from live~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected tod40085da631c,f77b7e46bd09,0a7f933d06e2. Known Issues refreshed: job-radar 403MODEL_NOT_IN_PLAN(streak 5), Telegram thread-delivery failures,weekly-save-restricted-bot-restartscript restored, residual pre-fix dispatch errors oncbfe434a3b5e/1a8e90ea2271. - Updated
systems/hermes-agent.md— version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/modelcustom→http://100.78.63.20:20128/smart-route; skills 330 → 341; cron count 29 → 30. - Updated
current-state.md—_config_version: 46and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section. - Updated
runbooks/job-radar.md— corrected primary model row (custom:omni-101 / gpt-5.6-terraviacommand-code) and added the 403MODEL_NOT_IN_PLANfailure mode with recovery steps. - Updated
systems/messaging-integrations.md— Telegramallowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup). - Updated
index.md— last-updated to 2026-09-30, cron count to 30, new incident listed. - No secrets written.
[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
- No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
- Updated
systems/scheduled-tasks.md— table and counts re-verified against live~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 —dca8482e4f76(403MODEL_NOT_IN_PLAN, ×5),1a8e90ea2271(HTTP 410, model retired upstream, ×7),fad5a86979d4(HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes:1b23c042e17dandc8d52904e1d6delivery targets nowtelegram:-1004321904721:102;c0a9fa314542flagged for a revoked Telegram bot token; bare-telegramtargets noted on0e753e9720fd,fad5a86979d4,066bb5191280. Known Issues restructured into Resolved / Open / Historical. - Updated
systems/messaging-integrations.md— new Delivery target semantics section: a baredeliver: "telegram"resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails withMessage thread not foundand the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs. - Updated
systems/model-providers.md— provider stack rebuilt from liveconfig.yaml: 5 custom providers (omni-1015,384 models,Omniroute:201285,398,litellm1,728,command-code71,guacano46);custom:omnirecorded as removed. Default model correctedresilient-free→smart-route. New Retired / Unavailable Models table (retiredqwen3.5:397b, plan-gatedgpt-5.6-terra, catalog-missingdeepseek-v4-flash) with the lesson that a raw model pin never self-recovers. - Updated
systems/mnemosyne.md— new Known Issue: gateway background-process notifications were being written into the person-attributed tier as[USER] [Anthony Martin]memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy. - Updated
systems/hermes-agent.md— MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins. - Updated
index.mdandcurrent-state.md— last-updated bumped to 2026-10-01 with a Recent Changes block. - Vault sync: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live underwiki/, not at the tree root, andindex.mdsits at the vault root. - No secrets written.
[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split
- New page:
incidents/2026-10-01-cliproxy-connection-auth-expired.md— job06a0a1c8704f(AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run withHTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard(providercustom:Omniroute:20128, modelcliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d,79f9bb41ecf5). - Updated
systems/scheduled-tasks.md— counts re-verified against live~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01:06a0a1c8704f(HTTP 401),0e753e9720fd(HTTP 400 — same NVIDIA model pin as the already-failingfad5a86979d4, confirming the shared-pin diagnosis),066bb5191280(delivery failed; report itself generated). Streaks advanced:dca8482e4f76×5→×6,1a8e90ea2271×7→×8,fad5a86979d4×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees. - Updated
systems/messaging-integrations.md— corrected a wrong page fact. The page previously stated that a badthread_idalways loses the message. The 2026-10-01 records show two variants:Message thread not found(hard failure, message lost — the 2026-09-30 case) andconfigured thread_id <n> for telegram:<dm> was not found; delivered without thread_id(a thread-less fallback send does occur, yet the run is still recordeddelivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegramjobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today". - Updated
systems/current-profile.md— full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no~/.hermes/profiles/default/directory (the default profile is the top-level config;opsis the only named sub-profile), the model default issmart-route(notfreellmapi-auto), the default provider iscustom→http://100.78.63.20:20128/ OmniRoute (litellm is now only the last fallback entry), and the config is_config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings. - Updated
current-state.md— new Recent Changes (2026-10-02) block; frontmatter dates bumped. - Updated
index.md— last-updated to 2026-10-02; cron line now notes the failing count. - Structural finding — the wiki exists as two trees and they have drifted both ways.
/home/hermes/workspace/agent-estate-wikiis canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is whatwiki-maintain.shcommits./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads (pages underwiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions —systems/ignis.md,systems/nextcloud.mdand threeincidents/pages exist only in the vault, whileraw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdandreports/communities-wa-social-media-audit-2026.mdexist only in canonical — so canonicalindex.mdcurrently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees. - No secrets written; all credential-shaped values in config remain redacted.
[2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
- Updated
systems/scheduled-tasks.md— fleet re-verified against live~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), failing set still 6 but the composition moved.1b23c042e17d(Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the identicalHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboardthat the cliprox incident predicted — making it 2 of the 3cliprox/terrajobs confirmed broken, which retires any "one-job fluke" reading.79f9bb41ecf5is the last untested job on that pin (next due 2026-10-04 18:00). Streak recount againstexecutions.db:dca8482e4f76×6→×7, but1a8e90ea2271×8→×7 — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). Resolved:1b23c042e17d's staledelivery_failedflag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). Downgraded: the GitHub Trending Radar revoked-token claim is now "unverified at last run" —c0a9fa314542's 2026-09-30 19:22 run completed and delivered. New Known Issue recording that a delivered notice is not a successful job (both cliprox failures readdelivery_outcome: deliveredon afailedstatus). - New page
incidents/2026-08-26-work-outlook-basic-auth-disabled.md— theworkOutlook mailbox has been unreachable since 2026-08-26: himalaya is configured for password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workdies withlogin not supported. TCP/TLS tooutlook.office365.com:993is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap. - Updated
systems/messaging-integrations.md— Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable,workunreachable; frontmatterupdated/verified_onbumped. Added the tooling consequence:email/mail-lookupresolves the account before searching, so work-mail requests hit this wall and must be reported as account unreachable, not "no results found". - Updated
systems/hermes-agent.md— skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to six jobs across three model faults plus one credential fault, and a second entry added for the stock-take job being down; theEmail: connectedbullet corrected. - Updated
current-state.mdandindex.md— new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped. systems/skills-index.mdregenerated 341 → 344 — including a new skill,fleet-wide-failure-triage(devops). This was done by hand: job1a8e90ea2271is still blocked on HTTP 410 (retiredqwen3.5:397b) and cannot self-recover.wiki-maintain.shitself is healthy and was run manually, also writingraw/stocktake/freshness-20261003.txt— recorded so the next reader knows the stocktake is on manual refresh, not on schedule.- Vault mirror: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live underwiki/;index.mdsits at the vault root. - No secrets written; all credential-shaped values in config remain redacted.
[2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
New pages
incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md—mnemosyne_recallhad been scoring keyword-only:fastembedwas absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 without raising an error. Repaired 2026-10-03; the runtime guard now asserts fastembed +FASTEMBED_CACHE_PATH+ model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).incidents/2026-10-03-ct450-hermes-update.md— CT450 (hermesagent) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (ssh root@100.96.100.82→pct exec 450 -- …) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
Updated
current-state.md— new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and themnemosyne doctor/sqlite_vecmisread gotcha.systems/mnemosyne.md— Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage jobb63ed5817212documented.systems/hermes-agent.md— version corrected tov0.21.5+6169.gab86a20(wasv0.21.5+3837.gbfda74c), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (noprofiles/default/;opsis the only named sub-profile), Known Issues bullet added.systems/memory-backend.md— Known Issues bullet for the FTS-only degradation.systems/scheduled-tasks.md— 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window),79f9bb41ecf5due 2026-10-04 18:00; bare-telegramdelivery warnings confirmed live.agents/hermes-production.md— version, Python, profile, provider routing, cron count and messaging refreshed; stalecustom:litellm/freellmapi-autoreferences replaced.agents/hermes-chaos.md— CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.infrastructure/hosts.md— CT450 row updated to v0.21.5.runbooks/update-hermes-safely.md— new "Second instance — CT450" section (access path + pre-flight checks).runbooks/provider-health.md— stale v0.19.0 version reference corrected.index.md— last-updated bumped; both new incidents linked; cron-fleet line re-dated.
Note — mnemosyne doctor on this host prints vec0: present_but_unloadable and marks the vector tiers unavailable. That is a Python-3.11-legacy-venv artefact, not the gateway's state. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py.
[2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.
- Updated
systems/scheduled-tasks.md— fleet re-verified against live~/.hermes/cron/jobs.json+hermes cron list: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 6 → 7. The one new failure event in the window is79f9bb41ecf5(Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identicalHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. It was the last untested job on thecliprox/terrapin, so that pin is now 3 of 3 confirmed broken — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271) and the Known Issues Open list all updated; frontmatter dates bumped. - Updated
incidents/2026-10-01-cliproxy-connection-auth-expired.md— new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still Open — needs the OmniRoute connection re-authenticated before any repin. - Updated
current-state.md— new Recent Changes (2026-10-05) block: the cliprox 3/3 settlement,1a8e90ea2271's repeat 410 on 2026-10-04, confirmation that no interactive sessions occurred in the window (all 10 active sessions were cron), and a streak-counting caveat. - Updated
index.md— last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed". - Streak-number caveat recorded:
hermes cron list's "N failures in a row" is thefailure_streakcounter injobs.json, not a DB count.~/.hermes/cron/executions.dbnow retains only rows from 2026-09-30 onward (1,001 rows total), soexecutions.dbrecounts undercount older streaks — the earlier DB-based "×7" for1a8e90ea2271undercounts against the live ×11. Quote the live counter and note the retention floor. - Vault mirror: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source —wiki/systems/scheduled-tasks.md,wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md,current-state.md,index.md,log.md. - No secrets written; all credential-shaped values remain referenced by connection ID only.
[2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0
Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.
New
incidents/2026-10-05-dead-model-pin-repin.md— the seven-job dead-model-pin repin. Records the four dead pins (cliprox/terraHTTP 401 ×3,ollama-cloud/qwen3.5:397bHTTP 410 ×1,nvidia/deepseek-ai/deepseek-v4-flashHTTP 400 ×2,command-code/gpt-5.6-terraHTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (free-stack×6,smart-route×1 oncustom:omni-101) over another raw model name.
Updated
systems/scheduled-tasks.md— Count block rewritten: 0 failing (was 7), with all 30failure_streakcounters at 0; four jobs carry a stale pre-repinlast_status: errorthat clears on their next scheduled run. Seven job-table rows updated (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271,fad5a86979d4,0e753e9720fd,dca8482e4f76), plusc0a9fa314542. New### Resolved 2026-10-05subsection in Known Issues; five resolved bullets removed from### Open, leaving bare-telegramtargets, the two-wiki-tree drift, the work-mailbox block, the stalecustom:omnikey and the delivered-notice rule as the standing open set.index.md— new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.current-state.md— new Recent Changes (2026-10-06) block; frontmatter dates bumped.
Findings worth the ink
dca8482e4f76(Daily Marketing Job Radar) delivered to topic 1030 at 23:56 — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for79f9bb41ecf5(23:39) and06a0a1c8704f(23:45).- GitHub Trending Radar's revoked-token diagnosis is dead.
c0a9fa314542ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. NogetMe→ 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright. delivery_outcome: deliveredstill does not mean the job worked. Every pre-repin failure recordeddelivered, because the failure notice is itself the deliverable. Judge on the executionstatuscolumn.- A stale notice opened the review. Anthony was prompted by a
85edf6772bcanotice that had sat in topic346since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice. - Window tally (
executions.db, 2026-10-05 00:00 → 2026-10-06 00:15): 185completed, 3failed(all pre-repin, 03:00–07:00), 2unknown(aborted first attempts of79f9bb41ecf5anddca8482e4f76at 23:33/23:48, superseded by their successful re-runs) and 1running(this audit).
Vault mirror: the new incident page plus systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
[2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.
Updated
systems/scheduled-tasks.md— Count block corrected: the stale pre-repinlast_status: errorset is now three (1b23c042e17d,fad5a86979d4,0e753e9720fd), not four —1a8e90ea2271cleared its own when it ran at 03:00. Added a Re-verified 2026-10-07 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176completed/ 0failed/ 1running; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.index.md— last-updated → 2026-10-07; cron-fleet line re-dated (0 failing as of 2026-10-07, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.current-state.md— new Recent Changes (2026-10-07) block; frontmatter dates bumped.
Findings worth the ink
1a8e90ea2271ran itself at 03:00 and proved the repin.wiki-stocktake-maintenancecompleted 2026-10-06 03:00:31 → 03:03:32 oncustom:omni-101/free-stack— its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retiredqwen3.5:397b). It regeneratedsystems/skills-index.md(344 → 346 skills), wroteraw/stocktake/freshness-20261006.txtand committed to Gitea as33f4e95. Its four HTTP-410 incidents (6df957a05ec6,d284168d3663,cda67de5b3e3,e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. The daily manualwiki-maintain.shrun can stop.- Vault mirror drift found and partly repaired.
systems/skills-index.mdin/home/hermes/wiki/agent-estate-wiki/was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item. - Window tally (
executions.db, 2026-10-06 00:00 → 2026-10-07 00:00): 176completed, 0failed, 1running(this audit) — 177 rows. 14 sessions, allcron; no interactive (non-cron) sessions, and no new incident opened.
Vault mirror: systems/scheduled-tasks.md, systems/skills-index.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
[2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings
Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.
Updated
systems/scheduled-tasks.md— new Re-verified 2026-10-08 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176completed/ 1failed/ 1running; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New Resolved 2026-10-07 Known Issues section, plus two new Open items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.index.md— last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.current-state.md— new Recent Changes (2026-10-08) block; frontmatter dates bumped.
Findings worth the ink
- The incident machinery had its first real workout since 2026-09-30 — and it worked.
62619aa6bcb3(DHL Tracking) failed its 08:00 run insideensure_browser()withTimeoutError: timed outon a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident62619a_f485ccfd4a0d(failure_type: timeout) was opened 08:00:52, alerted 08:00:58 and auto-resolved 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed. - The tracked parcel has arrived. The 14:00 run reported waybill
8052838382asDelivered, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed. - Stocktake ran automatically for the second consecutive day (
1a8e90ea2271, 2026-10-07 03:00, commit8c63ef2; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once. - Two real defects surfaced from the stocktake's own log. (1) Step 3 of
wiki-maintain.sh— the Quartz rebuild — is skipped on every run: there is noquartz.config.tsand nonpx, sopublic/is never refreshed and the published site silently ages. (2)skills-indexunder-reports by one: 347SKILL.mdfiles exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicateproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.md(the other duplicated basename,hermes-api-vault, survives because its two copies sit in different categories). Both are now Open items in systems/scheduled-tasks. - Unchanged, still carrying a stale
last_status: error:1b23c042e17d,fad5a86979d4,0e753e9720fd— due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-telegramtargets and thecustom:omnistale-key references are also unchanged.
Vault mirror: systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
[2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
Window audited: 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days.
Pages written
incidents/2026-10-09-cron-provider-timeout-storm.md— new.incidents/2026-10-09-memory-recall-relevance.md— new.systems/scheduled-tasks.md— Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.systems/mnemosyne.md— live tier counts refreshed; new dated counts line with a pointer to the recall incident.current-state.md— Recent Changes (2026-10-09) block; frontmatter dates bumped.index.md— last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.systems/skills-index.md+raw/stocktake/freshness-20261008.txt— the stranded 2026-10-08 03:01 stocktake artifacts committed.
Findings worth the ink
- The fleet had its worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight remain open. The watchdog (96eb95ce12f2) failed twice as a cascade — it exits 1 when it finds ERROR lines inagent.log, i.e. it reported the storm rather than suffering it. - The 2026-10-08 window was never audited — this job's own 23:59 run failed with the same timeout (incident
cbfe43_fa81052baaf7) and committed nothing. - The stocktake has a two-day gap. 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no
freshness-20261009.txtexists.1a8e90ea2271is the one job still atfailure_streak: 2. - An instruction was reported done and was not. Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that
62619aa6bcb3was "stopped and deleted". It is still enabled (paused_at: null) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit. - Memory recall relevance — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is not in live config (
memory.mnemosynecarries no recall-weight keys; no top-levelmnemosyne:key).
Vault mirror: the touched pages mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
2026-10-10 — Stocktake (manual run, wiki-maintain.sh)
systems/skills-index.mdregenerated — 346 skills / 70 categories (unchanged count; frontmatterupdated/verified_onrefreshed to 2026-10-10). Live on-disk count is 347SKILL.md; the one-skill gap is the known de-dup ofproductivity/obsidian-git-sync/obsidian-git-sync/.- Freshness audit →
raw/stocktake/freshness-20261010.txt: 62 stale (≥21d), 23 fresh, 7 no-date of 92 pages. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. Nofreshness-20261009.txtexists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands. current-state.md— new 2026-10-10 section. The 2026-10-09 timeout storm window re-tallied fromexecutions.db(2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163completed, 15failed— the prior audit recorded 162/15+1 running; the strayrunningrow is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).- DHL monitor re-probed, still enabled.
62619aa6bcb3—enabled: True,paused_at: null,0 8,11,14,17,20 * * *,last_status: ok. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. No cron mutation performed by this audit — retirement stays open item #1. infrastructure/tailscale.mdverified againsttailscale serve status— all 13 tailnet Serve routes plus the:8443/webhookFunnel match the page. No drift; not edited.- Not verified / left alone: the Quartz rebuild step (no
quartz.config.ts,npxcannot fetchquartz—public/was NOT refreshed, confirming the carried open item);docker ps(permission denied on/var/run/docker.sockfrom this session, soinfrastructure/docker-services.mdcould not be probed and is left stale); the 80d2026-07-22cluster (no live source to check against). - Sessions in the 2026-10-09 window (
state.db, epoch REAL): 19 total — 6cron, 11ios, 1talaria-ios, 1telegram. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — thedeliveredcount includes failure notices).
No secrets written; job/incident IDs only.
[2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-11T00:19). Day tally: 10-07 177/1,
10-08 178/2, 10-09 163/15, 10-10 174 completed / 0 failed. Non-clean last_status 10 -> 5.
Incidents: 7 closed during 10-10, 5 open (3 jobs). Open item #1 closed: the DHL monitor
62619aa6bcb3 is retired and verified absent from jobs.json by ID lookup (the 10-07 and 10-09
"deleted" claims were false).
Runtime/memory. The 2026-09-28 managed-runtime-dependency class recurred in the gateway env:
mnemosyne (+native deps) and elevenlabs were missing from the committed uv env
795020dbf24343cd9c52df24723dca83 that activate_dependencies() loads. Repaired 21:26-21:34 and
verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated;
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
New page. systems/paseo - Paseo daemon v0.11.2 on 0.0.0.0:6767 (user unit
paseo-daemon.service); recorded the daemon run vs daemon start supervisor trap, the
/usr/bin/paseo 0.7.2 vs daemon 0.11.2 version split, and the truncated-cline false provider fault.
Platform. Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 answers
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
Pages touched: systems/paseo.md (new), systems/scheduled-tasks.md, systems/hermes-agent.md,
systems/mnemosyne.md, systems/messaging-integrations.md,
incidents/2026-09-28-hermes-managed-runtime-deps.md, current-state.md, index.md, log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-11] stocktake | Daily maintenance + live drift corrections
- Ran
wiki-maintain.sh: skills-index regenerated (346 skills / 70 categories), freshness audit written toraw/stocktake/freshness-20261011.txt(62 stale / 24 fresh / 7 no-date of 93 pages) - OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed. Verified via
pct list,pct config,ssandtailscale status: CT600omniroute(100.88.81.19) is RUNNING and serving:20128; CT601omni(100.93.204.4) is STOPPED withonboot=0. Hermes resolvescustom:omni→omniroute:20128→ CT600, not CT601. systems/omniroute.md rewritten with the live node table. - Third OmniRoute node found:
omniroute-1at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provideromni-101(5384 models, defaultsmart-route, carriesskills_hub). Not inpct list— identity unconfirmed. - Ignis duplicate found: a second
ignis-ignis-1container runs on CT200 (imageignis-server:latest, host port 8092, vault/opt/ignis/vaults), separate from CT460'signis(port 8080, vault/home/hermes/wiki/agent-estate-wiki). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them. - Tailscale route table corrected against live
tailscale serve status:/agent-estatetarget is barehttp://127.0.0.1:8769(not.../8769/agent-estate-role-dashboard.html); previously-missing tailnet-only:9443→127.0.0.1:5173documented. CT201 routes refreshed (8085 targets +/steel5173 +/steel-api3000). - Proxmox inventory rebuilt from live
pct list+qm list: added CT475agentzeroand VM110omarchy; corrected 282ubuntu-server-xfceto stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). VM403open-clawand VM401hermes-oldare absent fromqm list— flagged unconfirmed rather than guessed. - Nanobot verified on CT333: gateway
nanobot-gateway(PID 2259048) listening on 8765/8766; NOT a systemdnanobot.service/gateway.service(both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified. - Docker services page corrected: the old "Docker runtime not available" note was a CT460 socket-permission artifact (
/var/run/docker.sockisroot:docker srw-rw----,hermesuser not in groupdocker, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo). - Flagged for follow-up, root cause not investigated:
mycompose-db-1on CT200 is inRestarting (1)— crash-looping. - Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.
[2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-12T00:10). Day tally: 10-07 177/1,
10-08 178/2, 10-09 163/15, 10-10 175 completed / 0 failed (finalised), 10-11 171 completed / 0 failed + 1 running — two consecutive clean days. Non-clean last_status steady at 5;
0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected).
Corrected a false alarm instead of logging a fault. The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a naive-UTC vs AWST comparison error — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in systems/mnemosyne and systems/scheduled-tasks so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.
New content. queries/retained-research-answers was a 2026-07-22 stub and is now a real page — the
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB;
-CompressLevel 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
was undocumented; the live datastore table plus host mount points are now in infrastructure/proxmox
(/mnt/ssd0 93 % / ~63 GiB free is the pressure point).
Stale labels fixed. incidents/2026-08-18-mnemosyne-capture-pipeline carried status: active from
August despite capture being verifiably healthy — now superseded, with the live evidence and a caution about
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
and the skills-index blurb no longer claims no regeneration since 2026-10-07.
Verified, not changed. The blocked-page-recovery lesson that session reported patching
(recovered: false = archive routes failed, not task failed; API-first rung) is present in the live skill.
Pages touched: systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/proxmox.md,
queries/retained-research-answers.md, incidents/2026-08-18-mnemosyne-capture-pipeline.md,
current-state.md, index.md, log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections
Script run. wiki-maintain.sh completed all three stages: skills-index regenerated from the live
skill tree (346 skills across 70 categories, unchanged count), freshness audit written to
raw/stocktake/freshness-20261012.txt (54 stale / 32 fresh / 7 no-date of 93 pages), and the Quartz
step took its documented no-quartz.config.ts path (build skipped, public/ unchanged — expected, not a
failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and
written its artifacts, so results were read from the report rather than assumed.
Drift corrected from live probes (2 pages):
- systems/nanobot — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on
CT333 is back up, HTTP
200onhttp://100.122.189.81:8766/. Port8900— the endpoint the disabled OmniRoutenanobotprovider connection uses — is still closed, sostatusstaysdegradedrather than flipping to active. SSH to CT333 is unreachable from CT460, sowhatsapp-bridge.servicewas not re-verified. - systems/home-assistant — carried
confidence: lowplus a page-wide "cannot confirm HA runs" caveat from 2026-07-22. Live: HTTP200onhttp://100.68.139.5:8123/, Tailscale nodehomeassistantactive and offering an exit node, VMID 100 running perqm list. Confidence raised low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460).
Verified and left stale. systems/openrouter and systems/nextcloud were probed shallowly and nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork.
Pages touched: systems/nanobot.md, systems/home-assistant.md, systems/skills-index.md,
raw/stocktake/freshness-20261012.txt, log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes
Window. 2026-10-12 00:00 → 23:59 AWST (sessions from ~/.hermes/state.db, cron from
~/.hermes/cron/executions.db + cron_incidents; live probes 2026-10-13 00:05–00:45). One former
compaction of this job left part of the review written but the log entry and cross-page updates
unfinished — completed here.
Headline 1 — a clean run hid a dead credential. c0a9fa314542 (GitHub Trending Radar — Weekly
Roundup, Mondays 09:00) reported completed with delivery_outcome: delivered on 2026-10-12, but its
fetch-then-send script died at digest.py send(). Live probe of the bot token in
/home/hermes/workspace/github-trending-radar/config.json returns HTTP 401 from getMe — the token is
a valid 46-character shape, i.e. revoked, not malformed. The fetch path is healthy (100 repos, 26 cache
files in 24 h). Critically, this supersedes the 2026-10-06 entry on current-state, which concluded
the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in
systems/scheduled-tasks: a completed run does not clear a credential fault unless the run exercised
the same call that failed, and green cron status is not delivery proof. (Credential was present; value
neither recorded nor repeated.)
Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new). The 2026-10-12 07:01 Daily
Marketing Job Radar run drove camofox-browser.service to 2.5G, swap fully consumed, available: 0B,
383 tasks against the drop-in ceiling MemoryHigh=2G / MemoryMax=3G / MemorySwapMax=1G. REST /tabs
and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at
07:23:50, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis:
workload exceeded the cgroup, not a leak — OOMPolicy=kill never fired.
Headline 3 — systems/browser-backend corrected against live probes. Prior revisions were wrong on
three counts: the REST API listens on :9377 only (:9093 closed, :9378 does not exist — REST and
WebSocket are the same service, not two), Camofox runs as a systemd --user unit on the Hermes host
(CT460), not CT450, and the containment drop-in was undocumented. current-state.md also still described
the old port model; both pages now match the probe.
Cron state. 2026-10-12: 174 completed / 0 failed / 1 running (this audit, cbfe434a3b5e, started
23:59:41) — third consecutive clean day; 2026-10-11 finalised at 172 / 0. Retention floor moved to
2026-10-07 08:15. Fleet 29 jobs (26 enabled / 3 disabled); non-clean last_status 5 → 3
(0e753e9720fd error, 1b23c042e17d error, 066bb5191280 delivery_failed). fad5a86979d4 (Facebook
Birthday Weekly Registry Sync) and dca8482e4f76 (Daily Marketing Job Radar) both cleared on 2026-10-12 —
the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50.
Incidents — open 5 → 2. Closed on 2026-10-12: fad5a8_a13bbc837e1c + fad5a8_c80b59a907f0 at
03:05:39 (the two deepseek-v4-flash HTTP-400 rows, cleared by the registry sync's clean run) and
dca848_23fcd2b18aaa at 07:33:41. Remaining: 1b23c0_c35703f637b8 (alerted — weekly job, cannot
clear before Fri 2026-10-16 08:00) and 0e753e_e7f8194a7d06 (detected since 2026-10-01 — the job has no
retained executions, so it cannot clear until 2026-11-01). → incidents/2026-10-09-cron-provider-timeout-storm
updated from "8 still open" to 11 of 12 closed with the full closure timeline; the pattern recorded
there is that each incident clears only on the next successful run of its own job, so weekly jobs carry
their rows for up to a week after the fault is gone.
Mnemosyne (2026-10-12 12:02 health run, green). Working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 % — both above the 95 % floor, no reindex; orphan embeddings 0; FTS5 exactly aligned;
0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → systems/mnemosyne.
Sessions. 18 in the window, all cron, zero interactive. The 03:00 stocktake 1a8e90ea2271 ran
clean and pushed 0eee5f1 (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70
categories — the known 347-vs-346 on-disk off-by-one persists).
Pages touched: current-state.md (new Recent Changes section + superseded-correction tag),
systems/scheduled-tasks.md (frontmatter, job inventory, non-clean statuses, re-verified history, five
job rows, ### Resolved 2026-10-12, corrected incident IDs, ### Open additions, new
## Re-verified 2026-10-13 block), systems/browser-backend.md (port + host + containment rewrite,
history entries), systems/mnemosyne.md (12:02 coverage), incidents/2026-10-09-cron-provider-timeout-storm.md
(status + full closure table + timeline), index.md (date, four blurbs), log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only.
Cron. 2026-10-13 = 167 completed / 1 failed (plus this audit's own row in flight → the day finalises
at 168). Tail: 10-10 175 / 0, 10-11 172 / 0, 10-12 175 / 0 — the first failed run since 2026-10-09
(which recorded 15). Retention floor moved to 2026-10-08 07:11 (MAX_TERMINAL_EXECUTIONS = 1000; 1,001 rows
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
126 / 1, partial).
Finding 1 — wiki-stocktake-maintenance (1a8e90ea2271) died on a dead opencode credential. 03:00:43 →
03:01:57, RuntimeError: HTTP 404: No active credentials for provider: opencode. Pinned
custom:omni-101:free-stack. New incident 1a8e90_d11698bd7275 (state: detected, failure_type: agent,
alerted_at: NULL — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
on the opencode path (10-08 HTTP 403: OpenCode's free tier can only be used from within OpenCode; 10-09
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
run. → incidents/2026-10-05-dead-model-pin-repin
Finding 2 — the failure stranded its own generated artifacts (now committed). Step 1 (skills-index) and
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
tree: systems/skills-index.md (modified, date-only: updated/verified_on 10-12 → 10-13; 346 skills /
70 categories unchanged) and raw/stocktake/freshness-20261013.txt (untracked; 48 stale / 38 fresh /
7 no-date of 93 pages, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit.
Consequence: the stocktake published nothing on 10-13 — and because the agent row is what commits, even a
perfect script run leaves nothing behind when the LLM turn dies.
Delivery. 159 suppressed / 9 delivered / 1 no-record. Delivered: Daily Marketing Job Radar,
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (0e753e9720fd,
066bb5191280, both Telegram thread_id 34795) are unchanged since 2026-10-01.
Mnemosyne. Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no reindex was
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic
1050/1056 = 99.4 %). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
bare; the delivery layer still suppressed it (delivery_outcome: suppressed) — the matcher tolerates wrapping
rather than requiring the literal token as a strict control. → systems/mnemosyne
Sessions. 15 in the window, all cron, zero interactive — third consecutive window with no interactive
session.
Fleet. 29 jobs (26 enabled / 3 disabled), pins unchanged; non-clean last_status 3 → 4 (cleared
fad5a86979d4 10-12 03:05 and dca8482e4f76 10-13 07:02; added 1a8e90ea2271); open incidents 2 → 3
(1 alerted, 2 detected, 8 closed, 66 resolved; 77 rows).
Pages touched: systems/scheduled-tasks.md (frontmatter, retention floor, non-clean list,
## Re-verified 2026-10-14), systems/skills-index.md (committed stocktake artifact), current-state.md
(new Recent Changes + date), index.md (date + two blurbs), log.md.
Vault mirror. The 2026-10-13 audit was never mirrored to /home/hermes/wiki (last mirror bb09581,
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus skills-index.md, whose vault
copy was still the 2026-10-08 version. The vault's own raw/stocktake/ still holds only the 2026-10-03 report.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
Ran wiki-maintain.sh (03:01:09 → 03:01:10 AWST): step 1 regenerated systems/skills-index.md
(346 skills / 70 categories), step 2 wrote raw/stocktake/freshness-20261014.txt
(48 stale / 38 fresh / 7 no-date of 93 pages), step 3 skipped the Quartz rebuild again
(no quartz.config.ts, no npx) so public/ is untouched — the known carried item.
Finding 1 — a second, unclustered Proxmox hypervisor exists: m93p (100.104.51.98), PVE 9.2.0
(kernel 7.0.14-12-pve). pvecm nodes is empty on both pve and m93p — no cluster — and
guest IDs collide between them (m93p CT221 is gitea; pve CT103 is cliproxyapi; CT100 and
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
hypervisor port :8006 returned only m93p and pve. m93p had been sitting in the Tailscale
node list for weeks as an unremarkable online peer, with no wiki page recording it as a
hypervisor. Live guests recorded from a single pct list / qm list pass.
Finding 2 — four guests the wiki placed on pve live on m93p. CT221 (gitea), CT241 → CT1241
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are absent from live pct list on pve,
yet every one of the four services answers. /etc/pve/lxc/ on pve contains only 260.conf.bak
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, they were never
on this host. The 2026-10-11 inventory was wrong and every audit since carried it forward.
Finding 3 — the tailnet inventory had drifted badly. The page listed 30 nodes; live
tailscale status reports 45 non-funnel nodes (plus 23 funnel-ingress-node IPv6 peers, now
noted and deliberately excluded). Fifteen nodes were missing, including m93p, pve,
omniroute-1, paseo, qwenpaw, rustdesk, vaultwarden, social-dashboard, openweb,
omarchybook, sftpgo, rclone, omni and runtipi. Several states were wrong too
(anthonys-mac-mini is online on a different IP; omarchy-macmini / omarchy-quattro are not
live peers; anthonys-iphone is online, not active). The CT460 serve routes re-verified
exactly as recorded — 13 routes plus :8443/webhook Funnel and :9443 tailnet-only — so that
table needed no change.
Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling: camofox-browser.service
now reports Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M) with
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
fault, as diagnosed last cycle.
Also verified, unchanged: Mattermost CT255 stopped (peer offline 24d); VM 282
ubuntu-server-xfce is now running, not stopped as recorded.
Cron (verified from executions.db / jobs.json, 03:05): 2026-10-13 = 168 completed /
1 failed (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress 23
completed / 0 failed with the 03:00 stocktake 1a8e90ea2271 still running at probe time. Open
incidents 3 — 1a8e90_d11698bd7275 (detected, 10-13 opencode 404), 0e753e_e7f8194a7d06
(detected since 10-01, next slot 2026-11-01), 1b23c0_c35703f637b8 (alerted since 10-09, next slot
Fri 2026-10-16 08:00). Non-clean last_status: 4 (unchanged). Fleet 29 jobs — 26 enabled /
3 disabled.
Pages touched: infrastructure/proxmox.md (two-host rewrite, m93p inventory, four-entry
correction, storage, VM 282), infrastructure/tailscale.md (45-node inventory + funnel-peer
exclusion note, frontmatter), infrastructure/network-map.md (m93p entry, frontmatter),
current-state.md (new Recent Changes block), index.md (proxmox/tailscale blurbs + header
warning), systems/skills-index.md (stocktake artifact), raw/stocktake/freshness-20261014.txt
(new), log.md.
Not verified, left as stale: the remaining 46 stale pages were not probed against a live
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
single-host in framing: systems/omniroute, infrastructure/hosts, systems/model-providers,
infrastructure/docker-services — m93p also runs its own omniroute (CT101), so the
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
[2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
Ran 2026-10-16 00:0x–00:3x AWST. Scope is two windows, not one: the 2026-10-14 23:59 Wiki Audit
(cbfe434a3b5e) failed on the fault below, so the 10-14 window was never covered.
Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).
- Root cause.
httpcore.ConnectError: [Errno 111] Connection refusedagainsthttp://100.78.63.20:20128— theomniroute-1node, providercustom:omni-101(confirmed inconfig.yaml: custom providername: omni-101,base_url: http://100.78.63.20:20128). It surfaced asopenai.APIConnectionError: Connection error.→RuntimeError: Connection error.— the exact string recorded on every failing execution row. 328Errno 111lines in the liveerrors.log, 504 across the rotations. - Impact. 10-14 = 160 completed / 12 failed; 10-15 = 151 completed / 24 failed (+1 in flight).
Against 10-12
175/0, 10-13167/1. 36 failure rows across 13 jobs, but only 24 are genuine model-call failures on 12 jobs — the other 12 rows are the Mnemosyne Watchdog reddening on a log sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15). - The finding that matters most — there is no failover. Verified from live
jobs.json: all 18 enabledagent-mode jobs are pinned tocustom:omni-101(smart-route×10,free-stack×6,deepseek-paid-smart×2). The only other model-capable pin,custom:omni(CT600,omniroute:20128), sits one530c89f2be0— which isno_agent: true;d40085da631cis disabled. Completing runs inside the window by provider: 210no_agent/ no provider + 1no_agentoncustom:omni. So zero model calls completed in 34 hours, and the fleet has no job that could even report the second node's health. - Correction made during this audit. A first pass read Evening Stand-Down Briefing's clean 18:45 runs
(both days) as evidence CT600 stayed healthy. Wrong: it is
no_agentand never calls a model. Recorded as a standing rule — never read ano_agentjob's green row as provider-health evidence — in the incident page and in systems/scheduled-tasks. - User-visible cost: silence. Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The weekly Perth New Restaurant was missed outright (next slot 10-22).
- Recovery, verified live (not assumed).
1b31187397a5completed 10-15 22:01:56 (streak 4 → 0);96eb95ce12f210-16 00:01:36;b63ed581721210-16 00:02:29;ea79a2107e3610-16 00:09:48. Probes 2026-10-16 00:1x:GET /v1/modelson100.78.63.20:20128→ HTTP 401 in 10 ms (up, auth required); same onomniroute:20128/100.88.81.19. - Fleet bookkeeping. Non-clean
last_status: 3 → 12 (9 new:08eb86f02d37,ecc1fc13655a,dca8482e4f76,c8d52904e1d6,06a0a1c8704f,85edf6772bca,0e519874d280,1a8e90ea2271,cbfe434a3b5e; three cleared in the window:b63ed5817212,1b31187397a5,ea79a2107e36). Open incidents 3 → 12 (9alerted/ 3detected). Long-standing and pre-storm:1b23c042e17d(Fri 08:00),0e753e9720fd(1st 09:00),066bb5191280(delivery_failed, dead Telegram thread 34795). - Mnemosyne. Both monitors sit on the failing node, so no coverage figures were refreshed for ~34 h; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected.
- Self-healing gap left alone deliberately. The 10-15 03:00 stocktake failure left
raw/stocktake/freshness-20261015.txtunwritten andsystems/skills-index.mdstamped 10-14. Its next slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the scheduled run. - Sessions. 36 sessions across the two days, all
cron, zero interactive — fourth consecutive window with no interactive session.
Pages touched: incidents/2026-10-14-omniroute-node-connection-refused-storm.md (new, 9,303 B);
systems/scheduled-tasks.md (frontmatter, Count current-state bullet, new Re-verified block);
systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter +
blast-radius note on the two monitors); current-state.md (frontmatter + new top section, Updated: line);
index.md (Last updated, new incidents entry, scheduled-tasks bullet); log.md (this entry).
Nothing removed or archived — no retired job, no obsolete system, so index.md structure gains one page.
Durable fix still open (needs Anthony's call): the single-node topology. All 18 model-calling jobs on
one OmniRoute node means this recurs on the next omniroute-1 blip — as it did on 2026-10-09. Options are
a second provider pin on a subset of jobs, or router-side failover. Not applied — that is a routing
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
[2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
Window: cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
2026-10-18 00:31 → 00:5x AWST. Sessions reviewed: 30, all cron, zero interactive — fifth
consecutive window with no interactive session, so every finding below is from live system state, not
conversation.
Summary
omniroute-1(custom:omni-101) has a second failure mode: 503 load-shedding. 9 model calls across 7 jobs died onRuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.between 10-16 08:00:37 and 10-17 03:00:28 (~19 h). The control plane stayed up the whole time (/api/health200 {"status":"ok"},/v1/models401— re-probed 10-18 00:33), so a green health endpoint is not evidence that the model path is serving. This is not a repeat of the 10-14 storm: that was reachability (Connection error., Errno 111), this was the router serving and refusing under load. →incidents/2026-10-16-omniroute-503-resource-pressure(new).- The burst ended on its own and the fleet is green again.
10-16 = 165 completed / 8 failed,10-17 = 162 / 5(all five before 03:01),10-18so far7 / 0; no failure row of any kind since 10-17 03:00. Every affected job re-ran clean on its next slot —b63ed581721210-17 06:01,08eb86f02d3710-17 08:31,ecc1fc13655a+ea79a2107e36both 10-17 12:02, each auto-closing its own incident. No change was made on the node; nothing was restarted or repinned. - Mosquito Tracker removed — fleet 29 → 28 (
1b31187397a5). Absent fromjobs.json(updated_at 2026-10-18T00:36:17+08:00), last execution 10-16 15:00 (completed, delivered). The three disabled jobs are unchanged, so 25 enabled / 3 disabled. Non-cleanlast_status12 → 8; only1b23c042e17d(error, streak 2, next slot Fri 10-23 08:00) is structurally stuck. - The watchdog double-counts.
96eb95ce12f2exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one sentinel. Read failures per job, not per row. - Second consecutive stocktake hole, again left alone deliberately. The 10-17 03:00 slot failed, so
raw/stocktake/freshness-20261017.txtis missing andsystems/skills-index.mdis stamped 10-16; 10-15 is missing too (storm). The 10-18 03:00 slot closes it — no manual regeneration, because a hand-run would race the scheduled one. - GitHub Trending Radar is dead at the delivery end. The Telegram bot token in its
config.jsonreturns HTTP 401 ongetMe(probe 10-18 00:35); the job's GitHub PAT still returns 200, so only the delivery credential is gone. The cron row stays green (last_statusfrom the 10-12 09:00 run,last_run_at 2026-10-12T09:00:12). Next exercise Mon 2026-10-19 09:00. Not repaired — a new bot token is Anthony's call, and the secret is not written here. - Incidents: 9 opened, 6 closed in-window; open 12 → 8 (5
alerted/ 3detected). The ledger files 7 of the 9 503 rowsfailure_type: unknowndespite the explicit message (the two script-exit rows are typedscript), sofailure_typestill cannot be used to triage. - Mnemosyne: both monitors rode the burst (
ea79a2107e3610-17 00:00,b63ed581721210-16 12:03), both clean on every slot since — including 10-18 00:00 (delivery_outcome: suppressed). No fresh coverage figures exist (they report to Telegram, not the ledger); last explicit read stays 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %), above the 95 % reindex threshold, so no reindex is due.
Pages touched: incidents/2026-10-16-omniroute-503-resource-pressure.md (new);
systems/scheduled-tasks.md (frontmatter + Count bullet + stocktake row + Known Issues entry);
systems/omniroute.md (frontmatter + new Recent Changes section); systems/mnemosyne.md (frontmatter +
new Re-verified section); current-state.md (frontmatter + Updated: line + new top section);
index.md (Last updated, new incident entry, scheduled-tasks bullet, skills-index note);
log.md (this entry). Nothing removed or archived — no page became obsolete in the window.
Bridge housekeeping (verified 2026-10-18): every page this run touched was byte-identical between the
authoring tree (/home/hermes/workspace/agent-estate-wiki) and the vault copy
(/home/hermes/wiki/agent-estate-wiki, pages nested under wiki/) before the edits, so the seven files
were written to the authoring tree first, committed and pushed (4137dd6), then copied into the vault —
sha256 matched on all seven. The vault-only pages (wiki/systems/ignis.md, wiki/systems/nextcloud.md)
and the vault's own wiki/raw/stocktake/ series were not touched; nothing was deleted and no one-way sync
was run. Drift found, not fixed: incidents/2026-09-02-sftpgo-outage.md exists only in the authoring
tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page.
Durable fix still open (needs Anthony's call): the single-node topology — all 18 model-calling jobs on
custom:omni-101 with no failover, now seen failing two different ways in five days. Options remain a
second provider pin on a subset of jobs, or router-side failover. Not applied — a routing decision, not
an audit one.
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
[2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST
Window verdict: one failure all day, and it was a failure mode the wiki had never recorded. Fleet
28 jobs (25 enabled / 3 disabled); 10-18 = 168 completed / 1 failed against 10-17 = 163 / 5;
10-19 so far 5 / 0 plus this audit. 11 sessions in the window, all cron — no interactive
sessions, so no user decisions arrived to document; every finding below is machine-observed.
- A model stream repeated itself and Hermes delivered the placeholder to Telegram.
ecc1fc13655a(FreshRSS Midday, 12:00) wrote "Your request was not processed. Send it again if you still want me to carry it out.", delivered it (delivery_outcome: delivered, 12:08:11) and then died at 12:08:25 onRuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.New page → incidents/2026-10-18-degenerate-stream-canned-substitute. The open question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue) instead of substituting text on the job's normal route? - Mattermost: cause found — CT255 is stopped, superseding the 10-11 entry's "cause not determined".
Reconnect loop unchanged (
attempt 471at 00:14:07,connect timed out after 30s, 5-minute cadence), peer offline 29d, and the host answers neither TCP:8065nor ICMP. Platform config still saysenabled: true, so the loop will keep logging until the container is started or the block retired. - New page: systems/push-notifications — the bridge is broken and has been for as long as the logs
go back.
status.json:connected: false,error: …connection refused (nothing listens there on this machine); the bridge dialsHERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net, which/etc/hostsresolves to127.0.1.1where nothing serves 443, while the gateway itself is healthy on0.0.0.0:9119(/→ 302) and Serve still maps the name to it. 23,469 occurrences of that string acrossagent.log(.1-3)+errors.log(.1-2). Fix documented, not applied. - Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.
working_memoryhad no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18 03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in the page); 1 unresolved conflict. The 24 h threshold inmnemosyne-health-check.sh:92will keep flagging idle weekends. - Stocktake landed and closed the two-day hole:
1a8e90ea2271exited 0 on 10-18 03:00,freshness-20261018.txtwritten, systems/skills-index regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (dd53e5f). - Carried forward, still open: the GitHub Trending Radar delivery token (
getMe→ 401, verified 10-18) — its next exercise is Mon 2026-10-19 09:00; the single-nodecustom:omni-101topology with no model-call failover; and the vault drift below.
Pages touched: incidents/2026-10-18-degenerate-stream-canned-substitute.md (new);
systems/push-notifications.md (new); systems/scheduled-tasks.md (frontmatter + Count bullet +
FreshRSS row + totals); systems/messaging-integrations.md (frontmatter + new Mattermost section);
systems/mnemosyne.md (frontmatter + new tier/health block); infrastructure/tailscale.md (frontmatter +
inventory header + three missing peers + mattermost age); infrastructure/proxmox.md (CT255 age);
current-state.md; index.md; log.md. Nothing removed or archived — no page became obsolete and no
one-way sync was run.
Bridge housekeeping + vault drift (verified 2026-10-19 00:2x). Of the ten files this run touched, six
were byte-identical between the authoring tree and the vault copy before the edits; two more
(infrastructure/tailscale.md, infrastructure/proxmox.md) differed only because the vault holds
older versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and
proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (wiki/
nesting) with sha256 verified per file.
Vault drift measured: against authoring HEAD, the vault holds 74 identical pages, 11 stale and
12 missing. Stale: infrastructure/network-map.md, infrastructure/tailscale.md,
infrastructure/proxmox.md, infrastructure/docker-services.md, systems/skills-index.md,
systems/current-profile.md, systems/home-assistant.md, systems/ignis.md, systems/nanobot.md,
systems/obsidian.md, incidents/2026-08-01-mnemosyne-capture-failure.md. Missing:
incidents/2026-09-02-sftpgo-outage.md (flagged by the 10-18 audit and still absent), the seven
agents/* pages, reports/communities-wa-social-media-audit-2026.md, CONTRIBUTING.md, plus this run's
two new pages (copied immediately after). The vault repo's own HEAD is still 9bacd8e mirror(2026-10-16)
with a working tree carrying uncommitted 10-17/10-18 changes, and it has no remote, so it cannot be
pushed. Not fixed — a full mirror pass is a bigger call than an audit's mandate; flagged here.
No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.
[2026-10-20] audit | Nightly wiki audit — omni-101 503 burst 2 (2026-10-19)
- Incident re-opened. incidents/2026-10-16-omniroute-503-resource-pressure went back to
status: active. A second, identicalHTTP 503 … resource pressureburst hitomniroute-1on 2026-10-19 — 9 model calls across 8 jobs (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 (2026-10-16 08:00 → 10-17 03:00). Whole day 159 completed / 13 failed; 2026-10-18 was 168 / 1. - Facts corrected from the source of truth (
~/.hermes/cron/jobs.json,executions.db) instead of being carried forward: the burst-2 table's job names, schedules and model aliases were wrong (85edf6772bcais Tech & AI Newsletter Digest (Analytical), not "Fringe Festival Plan";ea79a2107e36is the Mnemosyne Health Monitor, not a news digest;ecc1fc13655ais FreshRSS Midday News Briefing),08eb86f02d37fails at 08:31 (not 08:01, it is a30 8 * * *job), and 2026-10-17 reads 163 / 5, not 162 / 5. All eight failing jobs readprovider: custom:omni-101,model: smart-route— the burst-1 note's wider alias mix (deepseek-paid-smart,free-IA,dark) no longer matchesjobs.jsonand is now flagged in the page for confirmation rather than silently overwritten. - Incident ledger read properly for the first time. Unclosed rows = 13 (12
alerted/ 1detected), not the7the 10-19 entry recorded — the earlier figures used a narrower filter. A rawclosed_at IS NULLcount reads 12 at 10-19 00:20 and 13 now; the entry states the query so future entries compare like with like. 4 rows opened and 2 closed on 10-19, plusea79a2_747b27556fd0closed 10-20 00:04:24. - Pages updated:
incidents/2026-10-16-omniroute-503-resource-pressure.md(burst-2 table rebuilt fromjobs.json, new findings, status back to active),systems/scheduled-tasks.md(new current-count bullet, non-cleanlast_statusrewritten 7 → 12, Known Issues addition, 8 job-table rows corrected),systems/omniroute.md(new Recent Changes section),current-state.md(new Recent Changes section),index.md(scheduled-tasks and incident summaries, "Last updated" → 2026-10-20),log.md. - Nothing removed or archived — no page became obsolete and no one-way sync was run.
- Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 15 sessions in the window, all
cron— no interactive sessions, so there were no user-facing decisions to record. - No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only.
- Vault drift (measured —
sha256per file against/home/hermes/wiki/agent-estate-wiki): 76 identical / 15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512 "missing", but those are build artefacts underpublic/,package*.json,serve.pyandraw/stocktake/, which the vault deliberately does not carry.) All six pages touched by this audit were copied in with per-filesha256verification. The stale pages left unmirrored —infrastructure/docker-services.md,infrastructure/network-map.md,systems/current-profile.md,systems/home-assistant.md,systems/ignis.md,systems/nanobot.md,systems/obsidian.md,systems/skills-index.md,incidents/2026-08-01-mnemosyne-capture-failure.md— are authoring-tree edits made from other agents' sessions; a full vault mirror pass remains the open call. - Not re-run this audit: the
public/Quartz rebuild andwiki-maintain.sh. Authoring commits95fd7d3→766882e→8da6ddf→0248698, pushed to GiteaAnthony/agent-estate-wikiand confirmed at the remote.
2026-10-21 — nightly wiki audit (cron cbfe434a3b5e, window 2026-10-20)
- A clean day — and a failure this ledger cannot see. 2026-10-20 closed 168
completed/ 0failed(plus 1unknown), the first zero-failure day since 2026-10-12, ending a seven-day run. Theomni-101503 fault did not recur (last failure row 2026-10-19), so incidents/2026-10-16-omniroute-503-resource-pressure returns tomonitoring. Zero incident rows opened, 4 closed → unclosed 13 → 9 (8alerted/ 1detected). - New page: incidents/2026-10-20-gateway-unclean-exit-orphaned-run. The gateway died with no exit path at
2026-10-20 09:09:01 (last heartbeat 08:50:28,
pid 2496376,suspected_oom=False), orphaning the 08:30 Gmail Action Digest run (08eb86f02d37, claimed 08:31:03) asunknown— "owner exited before a durable terminal state". No delivery row for the slot, no incident row created, and the job table still shows the previous slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits since 2026-08-01, 5 of them clustered on 2026-08-14/15). - New page: incidents/2026-08-08-gitea-config-backup-push-dead.
e27da6c34bbbreadsokand its executions readcompletednightly while nothing has reached Gitea since 2026-08-07 11:00:47 — 75 unpushed commits,fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'. Second instance of the "green row, dead job" pattern after the GitHub Trending Radar token revocation. - Corrected the 2026-10-18 canned-substitute page. Its "second surface" section attributed the
interactive-session substitution to an
HTTP 404Model-not-found; the retained log evidence isRebuilt-message restart limit (3) exceeded→reason=rebuilt_restart_limit_exceeded … api_calls=1/150 tool_turns=0 response_len=0. The page now carries the verified trigger and flags the unresolved model disagreement (session pinminimax/minimax-m3.1-flash:freevsmodel=qwen3.5:397bin the turn-end line). The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered, and is recorded as outstanding in current-state. - Pages updated: current-state (new Recent Changes section), systems/scheduled-tasks (new count bullet
— non-clean
last_status12 → 9, config-backup row annotated, frontmatter), systems/current-profile (model default moved tonous/meituan/longcat-2.5-preview:free, nofallbackskey, pins unchanged), infrastructure/backups (the scheduled Gitea push has been dead since 2026-08-08), incidents/2026-10-16-omniroute-503-resource-pressure (status back tomonitoring+ dated update), incidents/2026-10-18-degenerate-stream-canned-substitute (trigger corrected),index.md, this log. - Fleet: 28 jobs / 25 enabled / 3 disabled (unchanged). 20 of 28 pin OmniRoute; the 8 unpinned jobs
moved upstream to
nouswith the global default change — the fleet now runs two upstream paths. 2 interactive sessions in the window (talaria-iostax-return work; thedesktopturn served the canned non-answer) — the first non-cron sessions after weeks of cron-only windows. - Nothing removed or archived — no page became obsolete, and no one-way sync was run.
- No secrets written. The gateway token pasted in the 2026-10-20 desktop session was not transcribed into any page; both the incident and current-state entries describe the request without the credential. Host names, IPs, job/incident IDs, ports and probe status codes only.
[2026-10-21] stocktake | Scheduled wiki maintenance (cron 1a8e90ea2271, 03:00 AWST)
- Maintenance script ran clean —
wiki-maintain.shcompleted all three steps: skills-index regenerated (346 → 347 skills; newaustralian-tax-return-reviewunder Finance), freshness audit written (raw/stocktake/freshness-20261021.txt), Quartz rebuilt (596 files) and deployed topublic/,hermes-wiki-serve.servicerestarted. - infrastructure/tailscale — node table rebuilt from live
tailscale status --json. The 2026-10-19 table had drifted again:octop(100.92.163.122) andspringroll(100.100.153.125) were missing entirely,openmausbot/openmausbot-vmwere described in prose but absent from the table body,omarchymachad gone offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on DNS name rather than the JSONHostName— the latter isct260for thefreshrssnode, and both OmniRoute nodes share the hostnameomniroute, which silently collides. Now 51 non-funnel nodes (50 IPv4 peers +hermes). Container states cross-checked against livepct listonpve(100.96.100.82) andm93p(100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. - reports/uber-eats — two missing monthly reports recovered. The generator
(
~/.hermes/cron/scripts/generate-uber-eats-report.py) wrote only to~/wiki/reports/uber-eats/(the~/wikirepo), never to the canonicalagent-estate-wikitree, so the 2026-08 and 2026-09 reports were missing from the published wiki. Root cause fixed: the script now writes to~/workspace/agent-estate-wiki/reports/uber-eats/and mirrors to the legacy tree. It also now emits YAML frontmatter — without it the pages landed in the "no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing months copied in and verified against~/.hermes/uber_eats_report.json(2026-09: $51.29 / 2 orders; 2026-08: $211.16 / 7 orders). The 2026-08 previous-month column was corrected — the original run had no comparison payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change. - incidents/2026-08-15-skills-index-drift and incidents/2026-09-02-sftpgo-outage re-verified against live
state and annotated. The skills-drift page named a
regenerate-skills-index.shthat no longer exists on disk (regeneration now lives insidewiki-maintain.shstep 1/3); the SFTPGo page confirmed CT299 running and the health endpoint returning HTTP 401 (auth required, service up) — no recurrence. - Still stale, left alone deliberately: reports/communities-wa-social-media-audit-2026 (54d) — its source
document
/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.mdno longer exists on disk, so the audit baseline cannot be re-verified against a live source. Flagged, not edited. - Fleet unchanged — 28 jobs / 25 enabled. This job (
1a8e90ea2271) isactive,last_status: ok,model: free-stack/provider: custom:omni-101, next run 2026-10-22 03:00. - No secrets written. Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes.
[2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the omni-101 pin
- New page: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. The Daily Marketing Job Radar
(
dca8482e4f76, 07:00 weekday,custom:omni-101/smart-route) failed 2026-10-21 07:40:48 after 40 m 36 s onRuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s— a string this fleet has never produced before, and not the 503 resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Sessioncron_dca8482e4f76_20261021_070022wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls); incidentdca848_014affeb6954isalertedand unclosed; it was the day's only failure (171completed/ 1failed). - Infrastructure: Hermes Agent updated to
0.22.3—hermes updateran 22:47:25 → 22:57:38 AWST (action9eed3a99e799347c691ea52a72e122a1), pre0.22.2/d923a5eb…→ commit5b147ab036…(detached), stamp built 2026-10-21T14:57:38Z. Stagesnapshot=skipped(disabled by updates.pre_update_backup (mode: off)), so the update had no snapshot and the off-box copy path is still dead. Gateway restarted via systemd (pid868087,phase: running, heartbeat 00:13 AWST), dashboard respawned by argv (:9119, pid869852). Advisories printed, neither actioned: rebuild the search index (hermes sessions optimize-storage) and Preparing pinned cua-driver… (no completion line in the log). The intermediated923a5ebrebuild was never recorded on the page, so the previously-notedv0.21.5+6169.gab86a20(2026-10-04) was already stale — corrected. - Pages updated:
systems/hermes-agent.md(version/upstream bullets rewritten, frontmatter dated),systems/scheduled-tasks.md(new current-count bullet; 10-21 bullet marked superseded),systems/omniroute.md(new 2026-10-21 section: three signatures, one pin),incidents/2026-10-16-omniroute-503-resource-pressure.md(window-close update; staysmonitoring),incidents/2026-08-08-gitea-config-backup-push-dead.md(2026-10-21 run: backlog 75 → 76,message_id 8449),infrastructure/backups.md(76 commits;pre_update_backup: falsenow has a concrete exercised example),current-state.md(new dated section),index.md(new incident entry, scheduled-tasks summary, date). - Ledger facts as recorded this window: fleet 28 jobs / 25 enabled / 3 disabled; non-clean
last_status9 (7error/ 2delivery_failed) with one swap —08eb86f02d37out (its 08:30 slot ran clean and08eb86_472c7b609c80resolved 08:37:21),dca8482e4f76in; 1 incident opened / 1 closed, unclosed stays 9 (8alerted/ 1detected); 12 sessions, allcron, no interactive sessions; 2026-10-20 settled at 169completed/ 1unknown(the orphan) — the 10-21 audit's "168 / 0" was taken before the 23:59 slot landed, and that correction is recorded in bothsystems/scheduled-tasks.mdandcurrent-state.md. - No secrets written. Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only.
[2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)
- Ran
~/.hermes/scripts/wiki-maintain.shafter the audit so the new incident page is actually published, rather than waiting for the 03:00 stocktake. systems/skills-index.mdregenerated from the live tree: 348 skills (was 347) across 74 top-level directories; section order now follows the script's category-priority list. Auto-generated — no hand edits.- Freshness report
raw/stocktake/freshness-20261022.txt: 1 stale / 95 fresh / 6 no-date of 102 pages. The only stale page isreports/communities-wa-social-media-audit-2026.md(55 d). - Quartz build + deploy: 273 HTML files into
public/;hermes-wiki-serve.servicerestarted 00:18:05 AWST (the build wipes and recreatespublic/, so the running server always needs the restart). - Verified live:
http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip→ 200, and all four of its wikilinks resolve (systems/omniroute,systems/scheduled-tasks,systems/hermes-agent,incidents/2026-10-16-omniroute-503-resource-pressure); the new entry appears twice on the served index.
[2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)
systems/scheduled-tasks.md— new## Countbullet for the 2026-10-22 window: 172completed/ 0failed(+ this auditrunning), zero failures for the second time in three days; fleet unchanged 28 jobs — 25 enabled / 3 disabled; five incidents closed in-window and none opened, so unclosed rows fall 9 → 4; non-cleanlast_statusfalls 9 → 5; 27 sessions (15cron, 3subagent, 6desktop, 2acp, 1oneshot). Previous bullet marked superseded;updated/verified_onbumped to 2026-10-23,review_afterto 2026-10-30.current-state.md— newRecent Changes (2026-10-23)block at the top: the 172 / 0 window, incidents 9 → 4 unclosed and non-cleanlast_status9 → 5, the 10-21 recount (168 / 1), the stocktake no-commit anomaly, the skills-index root fix, the Quartz deploy, and the session mix;updated/verified_onbumped to 2026-10-23,review_afterto 2026-10-30.- Correction recorded: a DB recount (by both
started_atandclaimed_at, AWST) puts 2026-10-21 at 168completed/ 1failed, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it (floor 2026-10-08); the delta is written into the page rather than silently changed. incidents/2026-08-15-skills-index-drift.mdreopened (status: resolved→active,reopened_on: 2026-10-23). Evidence: the 2026-10-22 stocktake (1a8e90ea2271) ran 03:00:28 → 04:38:19 (97 m 58 s, vs 16 m 01 s on 10-21), finishedcompletedwith deliverysuppressed, and committed nothing — its regeneratedsystems/skills-index.md(mtime 03:00,## .Archive (41)section, 347 total) was still uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo'swiki-maintain.sh(find-based, 347 with.Archive) and the stale~/.hermes/scripts/wiki-maintain.sh(rglob-based, 348 without). Live counts verified: filesystem 348, with symlinks 376, minus.archive/(41) andapple/(4) = 331, loader catalogue 329 enabled (34 hub-installed / 52 builtin / 243 local).- Root-cause fix applied: the repo's
wiki-maintain.shnow prunes*/.archive/*from the skill scan (the loader does not expose archived skills), prints the archived count as an explicit exclusion, and nameshermes skills listas the authoritative loadable figure.systems/skills-index.mdregenerated: 306 skills across 69 categories (was 347 counting the parked archive). Script fix and regenerated page committed together. - Quartz rebuild + deploy: 102 files parsed, 602 files emitted to
public/, deployed, andhermes-wiki-serve.servicerestarted 00:14:22 AWST. - Freshness report
raw/stocktake/freshness-20261023.txt: 2 stale / 94 fresh / 6 no-date of 102 pages — the long-standingreports/communities-wa-social-media-audit-2026.md(56 d) and, newly over the threshold,systems/model-providers.md(22 d, updated 2026-10-01). The latter needs a review pass against the live OmniRoute catalogue before its nextverified_on. - No secrets written. Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
[2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST
- New page:
systems/conduit.md— the Conduit dashboard plugin + iOS app (voice calls, cards, push). Plugin tree~/.hermes/plugins/conduit_push, version0.64.0(HEADeaa14be, no tags upstream), remotekaishi00/hermes-conduit-notifier; gateway on:9119; one registered device (iPhone, app2.0.0); state/DB files and sizes recorded. Where the wiki previously had no page for the phone surface at all. - New incident: incidents/2026-10-22-conduit-memory-start-timeout — three
Memory provider 'mnemosyne' start for Conduit timed out; abandoning the callwarnings (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 sMEMORY_TIMEOUT_Sin the plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree to a 60 s default (CONDUIT_MEMORY_TIMEOUT_Soverride) plus aTokenErrorguard that degrades instead of aborting; the dashboard restarted 18:54:43 and no abort line exists after it (two calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). The patch is uncommitted. incidents/2026-10-09-cron-provider-timeout-storm.md→status: resolved. Its last two rows (1b23c0_c35703f637b8,1b23c0_103cbe04a79b, both on1b23c042e17d) closed 2026-10-23 08:04 when that weekly job's Friday slot ran clean; the job readsokagain. All 12 rows closed — this incident no longer contributes an unclosed row anywhere.- Cron ledger: window 2026-10-23 closed 169
completed/ 0failed(+ this auditrunning) — the third zero-failure day in four; 2026-10-22 settled at 173 / 0. Fleet 28 jobs (25 enabled / 3 disabled), unchanged for a sixth audit. Unclosed incident rows 4 → 2 (c0a9fa_764558b5ae83alertedsince 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and0e753e_e7f8194a7d06detectedsince 10-01, next slot 11-01). Non-cleanlast_status5 → 4 (c0a9fa314542,0e753e9720fd,fad5a86979d4,066bb5191280). Pins unchanged (11omni-101/smart-route, 8 unpinned, 6omni-101/free-stack, 2custom:omni/smart-route, 1deepseek-paid-smart= this audit). 14 sessions: 13cron+ 1desktop(18:26:12, the Conduit phone-call session that exposed the fault above). systems/mnemosyne.md: new block recording the 10–45 s cold start and the consumer-timeout rule (a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold start itself was not optimised — only the budget was raised.- Also updated:
systems/scheduled-tasks.md(new Count bullet +## Re-verified 2026-10-24section),current-state.md(new Recent Changes block; 10-23 block marked superseded),index.md(date, newsystems/conduitand incident lines, refresh of the scheduled-tasks / skills-index / storm blurbs), and frontmatter added toincidents/2026-07-29-mnemosyne-relay.mdandincidents/2026-08-04-omniroute-free-deepseek-silent-stops.md— both sat outside the SCHEMA (rule 4: every page has frontmatter). - Freshness: 102 pages scanned at the last stocktake; 3 carry a
verified_onolder than 21 days —infrastructure/hosts.md,runbooks/provider-health.md,runbooks/update-hermes-safely.md(all 94 d, unchanged). Nothing removed or archived this cycle: no job retired, no system obsolete, no decision reversed in the window. - Note (hygiene): a second, older copy of this wiki sits at
/home/hermes/wiki/agent-estate-wiki(101 pages, 3 incidents, no git remote, last commitc798304"mirror(2026-10-22 maintenance)"). It is not served —hermes-wiki-serve.serviceservespublic/from this repo — so it is a stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than syncing it, since no script owns it. - No secrets written. Job IDs, incident IDs, table names, counts, paths and HTTP status codes only.
[2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
- Window 2026-10-24: 167
completed/ 0failed(+ this auditrunning); 2026-10-23 settled at 170 / 0 (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet 28 jobs (25 enabled / 3 disabled), 4 non-cleanlast_status, 2 unclosed incident rows (c0a9fa_764558b5ae83alertedsince 10-19;0e753e_e7f8194a7d06detectedsince 10-01) — nothing opened, nothing closed in-window. Pins unchanged. 14 sessions: 11cron+ 3desktop. - New incident page:
incidents/2026-10-24-conduit-call-request-refused.md. Anthony asked to be phoned three times on 2026-10-24 12:53–12:56 (Call me now to test calls.→Yes. You do→CAL ME); every reply asserted the agent cannot make calls, with 0 tool calls in the session.conduit_call_userexists, is registered for every session, is discoverable viatool_search, and has a bundled skill (calling-the-user); the call settings were permissive (enabled/when_asked/decides/alertsall true). Root cause: capability asserted instead of discovered — not a missing tool, and not one weak model (the free model's answer was repeated after a switch tosmart-route). - Correction to
incidents/2026-10-22-conduit-memory-start-timeout.md: its closing evidence (“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a false positive.conduit-calls.json → endsis a ring buffer of turn-ends that fire a watch (MAX_RECENT_ENDS = 64), and its two entries are the cron sessions of that night (Mnemosyne Embedding Coverage Monitor 00:01, Health Monitor 00:04).history— real call records — is empty, so no call has been placed since the fix and the 60 s budget is unexercised by a call. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43 restart. systems/conduit.mdrefreshed: version0.64.0/eaa14be→0.69.0/f000bf0(22 commits on); the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the uncommittedplugin_api.pypatch is now five releases behind HEAD; new section documenting theconduit_call_usertool, its gate conditions and rate limits; device/state figures re-verified (last_seen10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59); store 569.9 MB / 1,520,229 events / 3,028buckets— the 10-24 page's 308buckets/ 62windowswas a narrower grouping, corrected rather than read as a collapse.- Also updated:
systems/scheduled-tasks.md(new Count bullet +## Re-verified 2026-10-25section + a date-bucketing note),current-state.md(new Recent Changes block; 10-24 block marked superseded),index.md(date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22 line). - Freshness: not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
run: 102 pages scanned, 3 carry a
verified_onolder than 21 days (infrastructure/hosts.md,runbooks/provider-health.md,runbooks/update-hermes-safely.md— all 94 d, unchanged). They are the next cleanup candidates. Nothing removed or archived this cycle: no job retired, no system obsolete, no decision reversed in the window. - Method note recorded in
systems/scheduled-tasks.md:executions.dbstoresstarted_atwith a+08:00offset and SQLite'sdate()normalises to UTC, sodate(started_at)moves 00:00–07:59 AWST rows onto the previous day. Bucket days withsubstr(started_at,1,10). - No secrets written. Job IDs, incident IDs, table names, counts, paths and timestamps only.
[2026-10-25] stocktake | wiki-maintain.sh + drift correction (tailscale, homepage-dashboard, 2 stale pages)
- Script ran clean.
wiki-maintain.shcompleted all three stages at 03:02–03:03 AWST:systems/skills-index.mdregenerated (302 skills / 69 categories, 45 archived — the count fell from 306 because four skills moved to~/.hermes/skills/.archive/:courier-tracking-monitor,pve-vm-console-recovery,model-provider-comparison-research,live-application-verification— all four confirmed present in the archive directory), freshness audit written toraw/stocktake/freshness-20261025.txt, Quartz rebuilt (630 files) and deployed topublic/,hermes-wiki-serve.servicerestarted. Verified the served pages return 200 afterwards. - Freshness audit at run time: 2 stale / 99 fresh / 4 no-date of 105 pages. The two stale pages were both re-probed live and corrected (see below); after correction the audit reports 0 stale.
infrastructure/tailscale.md— node table rebuilt from livetailscale status --json. The 2026-10-21 table had drifted again:springroll-browser(100.109.221.91) was missing from the table entirely — added (online).omarchybook(100.107.9.100) andopenmausbot(100.91.214.91) have both gone offline (online → idle/offline 3d).openmausbot's m93p LXC (CT1264) is still running but no longer has thetailscalebinary on PATH, so the node itself has stopped connecting.- Every offline age was stale and was refreshed:
mattermost31d → 35d,omni64d → 68d,elementsynapse91d → 95d,ubuntu-server-xfce60d → 64d,cloudreve88d → 92d,anthonys-imac27d → 31d,cachyos47d → 51d,conduit44d → 48d,freshrss-new64d → 68d,omarchymac3d → 7d. homeassistantstill advertisesExitNodeOption: truebutExitNode: false— it offers exit-node capability but is not currently serving as one; the old "active; offers exit node" wording overstated the live state and was corrected.m93p(100.104.51.98) reads idle/offline 13m ago in the peer table, but the host is plainly reachable (bothpct listprobes and the SSH session succeeded during this audit), so it is recorded as a Tailscale-side liveness artefact, not a hypervisor outage.- Container identities confirmed the authoritative way this run:
tailscale ip -4executed inside each candidate LXC on bothpve(100.96.100.82) andm93p(100.104.51.98), because peer names and LXC names do not always match. CT209 (cloudreve), CT255 (mattermost), CT555 (elementsynapse) and CT601 (omni) confirmed stopped; CT600 (omniroute, live OmniRoute) and CT101 (m93p) confirmed running. - Funnel peer count re-checked: still 23
funnel-ingress-nodepeers with IPv6-only addresses (excluded). Total non-funnel nodes 51 → 52 (51 IPv4 peers +hermesitself).
infrastructure/tailscale.md+systems/homepage-dashboard.md— the CT201 route tables were wrong. Livepct exec 201 -- tailscale serve statusshows CT201 serving Tailscale on both:443and:8443, and the old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead" note was false: no Traefik container or process exists on CT201 (docker ps -aandps auxboth come back empty fortraefik), andss -ltnpshowstailscaledbound to100.96.244.39:443. Corrected routes:/→ 8085 and/mgr→ 8084 on both ports;/steel→ 5173 and/steel-api→ 3000 on:443only (/steelon:8443returns 502; on:443it returns 200). The old table also routed:8443straight tohttp://localhost:8082, which has no listener — the live target is 8085 (asset_proxy.py), which itself proxies to the dead 8082 and returns 502. Both pages corrected. The CT460 route table was checked and matched live exactly (12 paths, no drift).reports/communities-wa-social-media-audit-2026.md(was 58d stale) — re-verified, not drifted. Checked against the live Department of Communities pages: the remit wording, the ten portfolios, the 1 July 2025 Housing → Department of Housing and Works restructure, the five-minister structure and the WA Seniors Card carve-out all still hold; the Department site was updated 23 October 2026 and its recent announcements (carers, youth grants, forced adoption) match the content pillars recorded here. Audience metrics were not re-counted and are left as the 2026-08-28 point-in-time figures;review_after: 2026-11-26stands.incidents/2026-08-26-work-outlook-basic-auth-disabled.md(was 22d stale) — re-probed, still open.himalaya envelope list -a work -s 1reproduces the identical failure (cannot build IMAP client/login not supported). The tenant still advertises onlyXOAuth2and the account still authenticates by password, so no remediation has landed. Status staysopen;updatedandverified_onbumped to the verification date with the probe output recorded on the page.systems/scheduled-tasks.mdchecked and left unchanged. Live~/.hermes/cron/jobs.jsonreads 28 jobs — 25 enabled / 3 disabled, matching the page. All 28 job IDs are present on the page. Non-cleanlast_statusis still 4 (0e753e9720fderror,fad5a86979d4delivery_failed,066bb5191280delivery_failed,c0a9fa314542error streak 1) and unclosed incident rows are still 2 (0e753e_e7f8194a7d06,c0a9fa_764558b5ae83) — identical to the page's 2026-10-25 entry. No drift, nothing to change.- No secrets written. Job IDs, incident IDs, IPs, counts, paths and timestamps only.
[2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
- Fleet was quiet but one job was not. Live
executions.db: 2026-10-25 settled 172completed/ 2failed; both failures are08eb86f02d37(Gmail Action Digest), which also fired six times in the day (08:31:14 → 10:26:37 runaway at 1 h 55 m vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53). New signature on the 19:18:41 fire:Cron admission missing: the owner never admitted this fire. All six fires — failures included — carrydelivery_outcome: delivered. Neither failure created acron_incidentsrow; the page's open-incident count is unchanged at 2. → newincidents/2026-10-25-gmail-digest-runaway-and-refire.md - Root cause of the missing context compression found: the Nous Portal credential is dead.
Nous Portal runtime credentials unavailable: Invalid refresh token— 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04,agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. → newincidents/2026-10-22-nous-portal-refresh-token-invalid.md - ⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.
A 2026-10-24 19:06:10 switch note reads "model was just switched from meituan/longcat-2.5-preview:free to
smart-routevia omni-101"; live config confirmscustom:omni-101/smart-route(api_mode: chat_completions),config.yamllast written 2026-10-25 19:13. A six-provider top-levelfallback_providerschain is live again (headed by the deadnouscredential), so the 2026-10-21 "no fallback chain" record is stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → the fleet is a single upstream path again. Corrected onsystems/current-profile.md,systems/model-providers.md,concepts/model-routing.md,comparisons/model-routers.md,systems/omniroute.md. - Auxiliary routing captured. At 2026-10-25 19:13 all eight
auxiliary:slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved offnoustono-think/freeapi/autooncustom:omni-101withfree_only: trueand afreeapi/deepseek-v4-flash→kg/kilo-auto/freechain. ⚠ Masking, not fixing —nousremains first infallback_providers. New section onsystems/model-providers.md. systems/scheduled-tasks.md— new 2026-10-26 entry added; the 2026-10-25 entry marked superseded. Fleet unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-cleanlast_status4 → 5 (08eb86f02d37joined asdelivery_queuedwith an emptylast_error); pins unchanged.systems/conduit.mdre-probed live. Version0.69.2inplugin.yamlanddashboard/manifest.json(both), HEAD942ddb8(2026-10-24 15:35:42, "Pulse: list candidates, decide in code, journal them (#342)"), 0 behindorigin/main, 24 commits on since the0.64.0baseline, working tree still carrying the uncommitteddashboard/plugin_api.pypatch. The dashboard gateway restarted 2026-10-25 18:42:33, so the upstream commits are now live — the previous "not live" note is superseded.conduit-calls.json → historyis still empty;conduit_showran once on 10-25 14:37. Status onincidents/2026-10-24-conduit-call-request-refused.mdstays open (untested, not fixed).systems/messaging-integrations.md— CT255 stillstopped(Mattermost offline 36 d,tx 312 / rx 0); 268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not outage duration).systems/hermes-agent.md— gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04, 19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat.index.md— header date, the two new incident links, and rewrittenconduit,messaging-integrationsandscheduled-taskssummaries.- No secrets written. Job IDs, incident IDs, base URLs, counts, paths and timestamps only.
- Not changed:
current-state.md(its dated sections are historical records), and the deadcustom:omniprovider entry (still named by 2 jobs — Anthony's call, not the wiki's).
2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
wiki-maintain.sh ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, hermes-wiki-serve.service
restarted). The first freshness report listed 8 stale pages; each was then diffed against live state and
corrected where reality could be verified. A second script run then rewrote the same-day report with
0 stale / 103 fresh / 4 no-date, so the committed report matches the tree it ships with.
infrastructure/hosts.md— CT600 is live, not retired. Livepct liston both hypervisors shows CT600 running onpve(100.88.81.19) and CT101 running onm93p(100.78.63.20); CT601 does not exist on either host (pct config 601→ no such config), so the old "retired → CT601" note was wrong on both counts. CT333 (clawtest) moved from "unverified" to verified running. Both hypervisors now carry a verified row (pve100.96.100.82 / 17 LXCs;m93p100.104.51.98 / 19 LXCs). Config version 33 → 51; skills 66 → 302; cron "25 active" → 28 (25 enabled / 3 disabled); Xvfb display :102 → :99 (Xvfb :99 -screen 0 1600x1200x24). Theomniprovider entry was replaced by the two real ones —custom:Omniroute:20128→http://omniroute:20128/v1(6,480 models) andcustom:omni-101→http://100.78.63.20:20128(the live default provider). The authoring tree vs the vault mirror are now distinguished by path.agents/hermes-production.md— Tailscale IP corrected:100.96.100.82→100.118.5.51. The old value is the PVE hypervisor, not CT460. Version v0.21.5 → v0.22.3 (5b147ab036d6); stable v0.22.5 is available (hermes update --check). The dead 8082 port claim was removed (no listener; live ports are 9119 gateway + 8085 webhook). Default provider iscustom:omni-101, and the fallback chain was rewritten from live config — its first entry is the deadnouscredential, so a fallover tries a broken provider first. All eightauxiliary:slots areno-think/freeapi/autooncustom:omni-101(moved 2026-10-25 19:13). Cron 30 (27/3) → 28 (25/3) with the 5 non-cleanlast_statusnamed. Mattermost still unreachable — CT255stopped, peer offline 36 d.agents/hermes-chaos.md— version v0.22.3, and both CT450 and CT460 now run the same build (5b147ab036d6); the "3 commits behind" claim is superseded (0 behindorigin/main). Default model issmart-routeoncustom→http://100.78.63.20:20128, not the olddeepseek-flash-mixoncustom:omniroute. Cron "11 paused" → 25 jobs (13 enabled / 12 disabled), with 9 names shared with CT460 (8 disabled here,Important-mail monitorthe reverse).infrastructure/tailscale.md— node table rebuilt fromtailscale status --json(keyed onDNSName, IPv4-filtered) and container identity re-confirmed from inside each LXC viapct exec <vmid> -- tailscale ip -4on both hypervisors.octoagent(100.117.10.68) andomarchymac-1(100.126.173.115) were missing entirely and are now listed — 52 → 54 non-funnel nodes (53 live IPv4 peers +hermes).m93pnow reads online (it was a stale peer record). Every offline age was refreshed (mattermost35d → 36d,omni68d → 69d,elementsynapse95d → 96d,ubuntu-server-xfce64d → 66d,omarchymac7d → 8d,omarchybook3d → 4d,openmausbot3d → 4d). Theomnirow now reads "CT601 removed from both hypervisors" rather than "CT601 stopped". Serve tables re-probed per host: CT460 matched live exactly (12 paths on :443 plus the :8443 Funnel and :9443 route); CT201 still 502s on/(asset proxy → dead 8082) while/mgrand/steelreturn 200 — the 502 claim was re-confirmed, not merely carried over.systems/memory-backend.md— re-verified against liveconfig.yaml: addedreflect.disabled_for_cron,sleep_threshold: 50,nudge_interval: 10, and live totals (2,350 memories — 842 working / 1,090 episodic / 418memoria_facts). Embedding coverage re-measured: 1,899 / 2,350 (80.8 %) overall, working 815 / 842 (96.8 %) and episodic 1,084 / 1,090 (99.4 %) — both tiers above the 95 % reindex threshold, so no reindex was triggered.runbooks/provider-health.md— the step-3 OmniRoute snippet was broken and raisedIndexError: the provider was renamedomniroute→Omniroute:20128and the key moved from an inlineapi_keyto akey_envindirection. The snippet now resolves the key fromkey_env. A new step 3b documents theomni-101quirk — its/v1/modelsreturnsInvalid API keyeven with the correct key while/v1/chat/completionsworks (asmart-routecall returnedminimaxai/minimax-m3.1-flash), so a models check on that node is a false failure. Also corrected the typo "config slocated" → "located". Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI/health200, opencode/nous/ollama fallbacks all 200.runbooks/update-hermes-safely.md— added the managed-runtime dependency re-assert step (hermes-runtime-deps-guard.sh) after an update, since the updater can leave a runtime withoutfastembed/dotenv/mnemosyne. Clarified--branch mainvs the stable channel (currently v0.22.5, local v0.22.3). Version andlast_testedbrought to 2026-10-26.incidents/2026-10-03-ct450-hermes-update.md— added a Current state section: both instances now run v0.22.3 (5b147ab036d6), CT450 is 0 behindorigin/main, and the CT450 Mattermost timeout is still an infrastructure condition (CT255stopped), not an update regression.incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md— added a Current state section: the follow-up is satisfied —fastembed0.8.1 imports in both managed runtimes (19da945341ad4794ac2bc8ff9fdd3643,d41b16c8b6674454a4a819bd4acced1f) and 0.8.0 in the legacy venv,FASTEMBED_CACHE_PATHpinned. Current coverage figures recorded; both tiers above the 95 % threshold, so monitor jobb63ed5817212(last_status: ok) has nothing to fire on.- All 9 corrected pages verified at
http://127.0.0.1:9120/<path>— every one returns 200, and the servedinfrastructure/tailscalecontainsoctoagent,omarchymac-1and54 non-funnel. - No secrets written. Job IDs, incident IDs, base URLs, key env var names, counts, paths and timestamps only.
Vault mirror drift (2026-10-26, measured not eyeballed)
The vault at /home/hermes/wiki tracks 125 files under two conventions:
agent-estate-wiki/<path> (28 files — the Obsidian-visible tree the recent audit
commits write) and agent-estate-wiki/wiki/<path> (93 files — the fuller mirror).
This run forward-synced every page it touched into both conventions where the
directory existed — 22 copies, all sha256-verified against the authoring source
before the vault commit. Two vault copies differed from the authoring tree
(infrastructure/tailscale.md, systems/skills-index.md under wiki/); both were
confirmed to be older versions (2026-10-21 vs authoring HEAD 2026-10-25), so
no direct vault edits were at risk.
Remaining drift (left for an explicit request — a full mirror pass is bigger than an audit's mandate):
| Convention | match | stale | missing |
|---|---|---|---|
agent-estate-wiki/<path> (top-level) |
25 | 2 | 80 |
agent-estate-wiki/wiki/<path> |
34 | 58 | 15 |
The two top-level stale files are current-state.md and
reports/communities-wa-social-media-audit-2026.md — both dated historical records
rather than drift. The vault repo is local-only (no remotes); it was committed,
not pushed.
[2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
- A planned gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
this time it was loud.
gateway.log:Received SIGTERM as a planned gateway stopat 14:33:35 (under_systemd=yes,parent_name=systemd,loadavg_1m=2.15). The drain then loggedShutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True→interrupting remaining work, and at 14:34:29Skipping .clean_shutdown marker — drain timed out with interrupted agents. The victim wasea79a2107e36(Mnemosyne Health Monitor), claimed 12:01:13 — 2 h 34 m in. It closed 14:35:59failedon a string this wiki had not seen:RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership. Unlike the 10-20 orphan it raised twocron_incidentsrows (14:33:37ea79a2_c3ab2d784ba3"Cron execution unverified; reconcileadmissions/724a69108b…json"; 14:35:59ea79a2_a4661554a3c2, the RuntimeError) and delivered the failure. → newincidents/2026-10-26-planned-restart-orphaned-cron-run.md - The Mnemosyne Watchdog did its job.
96eb95ce12f2red-flagged twice (14:46:12, 15:01:15,Script exited with code 1) reacting to the same logERROR, then self-resolved at 15:16:12 when the line aged out of its 15-minute window. Both rows (96eb95_0c89bd373a0a,96eb95_e32174134546) are closed. - Window tally and fleet: 2026-10-26 settled 169
completed/ 3failed, and all three failures are this one interruption (Health Monitor + the two watchdog reds); 2026-10-25 settled 172 / 2. Fleet unchanged: 28 jobs, 25 enabled / 3 disabled. Non-cleanlast_status5 → 4 (c0a9fa314542and08eb86f02d37cleared took;ea79a2107e36joins aserror, streak 2). Unclosed incident rows 2 → 3. Pins unchanged. One gateway start (14:35:18, teardown 54.06 s). 17 sessions in window (15cron, 1ios, 1telegram). - One incident row closed — the 2026-10-16 503 burst is fully shut.
c0a9fa_764558b5ae83auto-closed 09:13:00 when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on 10-25 or 10-26. Status staysmonitoring(the single-node exposure is unchanged). Updatedincidents/2026-10-16-omniroute-503-resource-pressure.md. - ⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.
(a) The radar's 07:00 slot closed
completedat 09:10, but its delivered text read “RUN INCOMPLETE — tool-call budget exhausted” — no Notion writes, no cover letters — so the day's backlog was executed by hand. (b)fad5a86979d4's 03:08 slot completed but endeddelivery_failed:Thread 34795 not found— the same dead thread the gateway's own 14:33:37 shutdown notice hit. →runbooks/job-radar.md - The radar's cover-letter pipeline changed the same morning. The PDF is now printed from the canonical
rendered HTML via Chromium (
build_cover_letter_pdf_from_html.py, created 10:37) instead of ReportLab, so a download matches the on-screen letter; the generation default moved todeepseek-paid-smart(10:47) andjob_dashboard_server.py's regenerate timeout went 120 s → 600 s (10:48, server restarted). →runbooks/job-radar.md - Files: new
incidents/2026-10-26-planned-restart-orphaned-cron-run.md; updatedincidents/2026-10-20-gateway-unclean-exit-orphaned-run.md,incidents/2026-10-16-omniroute-503-resource-pressure.md,runbooks/job-radar.md,systems/scheduled-tasks.md,current-state.md,index.md,log.md. Nothing removed or archived. No secrets written. - Unresolved: why a planned stop bypasses the drain deadline (
timed_out=Trueat 1.35 s of a 30 s budget); whetherfailure_streakshould count the unverified row (one interruption currently reads streak 2); and theThread 34795delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the unthreaded chat.
[2026-10-28] audit | Nightly wiki audit — window 2026-10-27 00:00 → 2026-10-28 00:2x AWST: the radar's tool-call budget ran out a second time — and this time the run is recorded as failed
- Window 2026-10-27: 172
completed/ 1failed(+ this auditrunning). The failure is the radar's budget overrun, and it is the story of the night → new page incidents/2026-10-27-job-radar-budget-exhausted. - The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
recorded as
failed.dca8482e4f76ran 07:01:08 → 11:03:54 (4 h 2 m 46 s), closedfailedand delivered "INCOMPLETE (tool-call budget exhausted before Notion write stage)": discovery, verification, dedup and classification completed; Notion ingestion, cover-letter generation and the final cover-letter audit never ran. Notion DB0f90ba2b-8b10-4d02-a62c-6f692d5b1168held 451 entries with 0 added for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88). Incident rowdca848_e93a538ade6bopened 11:03:53,alerted11:03:59, still open at this audit (failure_type: auth— which does not describe a budget fault). The previous day's identical overrun closedcompletedand was found only by reading the delivered text — that is why the 10-27 row matters. - Response, same day: the radar is now two jobs. Stage 1
dca8482e4f76Daily Marketing Job Radar - Discovery (0 7 * * 1-5,deliver: local, failures → Telegram topic 1030) doesDISCOVERY → VERIFY → DEDUP → SCORE, writes a handoff file and triggers stage 2. New stage 25fc292ff0f3bDaily Marketing Job Radar - Write and Letters (30 11,14 * * 1-5, Telegram topic 1030, created 2026-10-27T17:34:15) lands the Notion entries, cover letters, missing-letter audit and report. Two stage-2 test fires on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) bothcompleted, deliverysuppressed,last_status: ok, streak 0. Pins unchanged (custom:omni-101/smart-route). Fleet 28 → 29 jobs (26 enabled / 3 disabled). →runbooks/job-radar.md - The 2026-10-26 orphan class did not recur; its two incident rows closed.
ea79a2_a4661554a3c2andea79a2_c3ab2d784ba3auto-closed 2026-10-27T00:38:32+08:00 — unclosed rows fleet-wide 3 → 2 (the 2:0e753e_e7f8194a7d06Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the newdca848_e93a538ade6b). No 10-27 execution row carries the orphan strings and the watchdog alert log has no 10-27 entry. →incidents/2026-10-26-planned-restart-orphaned-cron-run.md - Non-clean
last_status4 → 5 (churn):ea79a2107e36cleared took;dca8482e4f76joined aserror(streak 1) and08eb86f02d37re-joined asdelivery_queued. Also non-clean:0e753e9720fd(error),fad5a86979d4(delivery_failed),066bb5191280(delivery_failed). - Two config corrections from live probes.
tts.providerisopenaipointed at the Mac mini speech server (https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech, modelphonon, voicealloy) — the page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked superseded rather than deleted). Live voice usesvoice.speech_server.url=wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/;GET /v1/pool→{"size":2,"in_use":0}, both unitsidle, while/v1/modelsreturns{"detail":"Not Found"}.voice.auto_ttsis stillfalse, andwake_word.enabledistrue. →systems/hermes-agent.md,systems/conduit.md systems/skills-index.mdwas regenerated by the 2026-10-27 03:00 slot but never committed — the authoring tree was dirty withupdated: 2026-10-27and a 302 skills / 69 categories total (against index.md's stale 306 / 70 of 2026-10-23). Freshness at that run: 0 stale / 104 fresh / 4 no-date of 108 pages (freshness-20261027.txt). Committed, andindex.mdcorrected to 302.infrastructure/tailscale.mdpeer table re-verified againsttailscale status --json; the node-count header and any state changes are recorded in that page.- Files: new
incidents/2026-10-27-job-radar-budget-exhausted.md; updatedincidents/2026-10-26-planned-restart-orphaned-cron-run.md,runbooks/job-radar.md,systems/scheduled-tasks.md,systems/hermes-agent.md,systems/conduit.md,infrastructure/tailscale.md,systems/skills-index.md,current-state.md,index.md,log.md. Nothing removed or archived. No secrets written. - Unresolved: whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
failure_type: auth; and the 10-27 radar incident row will only close when a clean stage-1 slot runs.
[2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted
- New page: incidents/2026-10-28-job-radar-stage1-iteration-cap — stage 1 (
dca8482e4f76) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closedfailed,failure_streak: 2, on "hit the agent iteration cap before the handoff could be written". No handoff file existed (~/.hermes/radar-ref/handoff/held only2026-10-27.json), so both stage-2 slots (5fc292ff0f3b, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43)completedwithdelivery_outcome: suppressed— the no-op recovery rule proven in production. Second consecutive stage-1 failure at a different ceiling (tool-call budget on 10-27 vs iteration cap on 10-28). - Updated: incidents/2026-10-27-job-radar-budget-exhausted (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did not close, unclosed rows 2 → 4) · runbooks/job-radar (10-28 slot write-up; "first real cycle" marked as run-and-empty) · systems/scheduled-tasks (new 2026-10-29 window bullet; prior bullet demoted to Prior window; the two in-window job rows refreshed) · systems/browser-backend (dated correction: Camofox restored,
browser-steelplugin removed, endpoint dark,STEEL_BASE_URLresidue in 10 live processes) · comparisons/steel-vs-camoufox · decisions/rollback-steel-to-camofox (2026-10-29 update) · systems/skills-index (stocktake-run note) ·current-state.md(new 2026-10-28 section) · index. - Stocktake folded, not paged:
1a8e90ea2271ran 03:00:28 → 03:37:58, closedfailed(streak 1, deliverysuppressed) onRuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203after regeneratingsystems/skills-index.md(302 skills) andraw/stocktake/freshness-20261028.txt(2 stale / 103 fresh / 4 no-date of 109 pages); it left both uncommitted and filed incident1a8e90_f57ab4cafab8in statedetectedwith no alert. The audit committed the two artefacts, clearing the dirty tree. - Live probes (2026-10-29 00:3x AWST):
jobs.json29 jobs / 26 enabled / 3 disabled, 5 non-cleanlast_status, pins 12smart-route/ 6free-stack/ 7 unpinned / 1deepseek-paid-smart;executions.db— 2026-10-28 settled 168 / 2 / 1 running, 2026-10-27 now 173 / 1, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP:9223and REST:3000refuse, tunnel/steel502,steel.martinwa.orgunresolved; no Hindsight service on this box (:8888closed, nohindsightunits) so systems/memory-backend and comparisons/memory-backends were left unchanged. - Removed/archived: none. Secrets written: none.
- Unresolved: the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days);
1a8e90_f57ab4cafab8sitsdetectedwithout an alert; both radar incident rows close only on a clean 07:00 weekday slot.
[2026-10-29] update | Steel Browser fully removed from the estate — Camofox is the sole backend
- Root cause of the long-standing
cloud_provider: steelresidue: the 2026-07-30 rollback disabled thebrowser-steelplugin and setbrowser.engine: camofox, but leftbrowser.cloud_provider: steelin~/.hermes/config.yaml. Every browser tool call then routed through_instantiate_explicit_cloud_provider()and failed with "no registered browser plugin has that name: steel". Becauseenginewas already correct, the split config was inert-but-wrong and survived ~3 months of audits. - Hermes (CT460):
browser:block now reads exactlycloud_provider: camofox,camera_provider: camofox,engine: auto,cdp_url: "". Thebrowser-steelplugin directory was deleted, and thebrowser-steelkey was removed from bothplugins.disabledandplugins.entries.STEEL_BASE_URLwas dropped from~/.hermes/.env. Hermes core source contains zero Steel references, so there is no Steel fallback path in the tree — returning to Steel now means redeploying it. - CT201 (runtipi): the entire Steel deployment was torn down — containers
steel-browser-api-1/steel-browser-ui-1, thesteel-networkDocker network, the imagessteel-browser-api:fixedand both ghcr.io Steel images, and the project dir/root/steel-browser(~6 GB reclaimed; disk 70G → 64G used). The Tailscale Serve routes/steel(5173) and/steel-api(3000) were deleted withtailscale serve --set-path <path> off; CT201's remaining routes are only/→127.0.0.1:8085and/mgr→http://127.0.0.1:8084, on both:443and:8443. Nothing listens on 3000, 5173 or 9223. - CT333 (clawtest): a second, separate and orphaned Steel install was found and removed — the
steel-browsercontainer (ghcr.io/steel-dev/steel-browser:latest, 5.53 GB, created 2026-05-30, publishing port 3000) and its image. It had no Serve route, no entry in~/.zeroclaw/config.toml(which usesbackend = "agent_browser"on 127.0.0.1:9515), no reference from Nanobot's config or the CLI-anything browser harness; its last CDP navigation was 2026-10-20, it received no requests after 2026-10-21, and its page crashed 2026-10-22. - Estate-wide sweep: every LXC on the PVE host was checked — no Steel containers, images, networks or listeners remain anywhere. Homarr, the homepage dashboard and the runtipi app configs had no Steel references.
- Gateway restarted 01:21 AWST to clear
STEEL_BASE_URLfrom the inherited process environment; the new gateway tree (PID3913841) readsSTEEL_BASE_URL=0, and Telegram, Discord, ntfy, email and Home Assistant all reconnected cleanly. The dashboard unit andagentmail-webhook.servicestill carry the variable in their inherited environment, but no core code reads it. - Reversibility preserved: CT201's compose file and Dockerfile are archived under
/root/_archived-steel/, plus/root/_archived-steel/steel-cache-archive-20261029.tgz; the images remain re-pullable from ghcr.io. Backups:config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z,config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z,.env.bak-pre-steel-cleanup-20261028T220947Z. - Camofox verified healthy after every change: REST
http://100.118.5.51:9377(andhttp://localhost:9377),/health→ 200,/tabs→{"running":true,"tabs":[]}, and live navigation confirmed working. - Pages corrected (15): systems/browser-backend · decisions/rollback-steel-to-camofox · decisions/index · comparisons/steel-vs-camoufox · comparisons/camoufox-vs-chrome · infrastructure/docker-services · infrastructure/tailscale · infrastructure/hosts · systems/homepage-dashboard · concepts/browser-automation · runbooks/recover-docker-service · current-state · index · agents/nanobot · systems/scheduled-tasks.
- Corrects the 2026-10-29 00:3x audit entry, which described Steel as merely "not reachable / endpoint dark". It is removed, not down. Per SCHEMA rule 9 the superseded claims are marked, not deleted.
- Removed/archived: none. Secrets written: none (variable names only; no key or URL values).
- Unresolved: the obsolete CT333 line in Nanobot's Obsidian vault (
Areas/Homelab/architecture.md, "Steel Browser | Clawtest | http://localhost:3000") still stands — that vault is outside this wiki and was not modified. Also noted, unrelated to Steel:Browser Use viewing unavailable: unsupported Hermes browser resolverhas logged in the gateway since 2026-10-11.
[2026-10-29] stocktake | General Agent Estate Wiki maintenance — Tailscale table re-verified, and a 23-day-old "never re-authenticated" claim turned out to be false
- Script:
wiki-maintain.shran clean (exit 0).systems/skills-index.mdregenerated from the live skill tree — 302 skills in 69 categories, with a new note that 45 skills are parked under~/.hermes/skills/.archive/and excluded from the count. (303SKILL.mdfiles exist outside.archive/, butproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.mdis a nested alias duplicate ofproductivity/obsidian-git-sync/SKILL.md, so 302 unique skills is the correct figure.) Freshness report written toraw/stocktake/freshness-20261029.txt; Quartz rebuilt (110 pages → 644 files) and deployed topublic/. - Freshness report: 2 stale / 105 fresh / 3 no-date of 110 pages. The 2 stale were
incidents/2026-10-01-cliproxy-connection-auth-expired.mdandincidents/2026-10-05-dead-model-pin-repin.md, both at 23d. - Updated infrastructure/tailscale — State column rebuilt from
tailscale status --json; 11 rows changed. Two were state transitions:sftpgo(CT299, pve) is back online (the 2026-10-28 audit had recorded itoffline <1d;tailscale ping→ pong in 0s,systemctl is-active sftpgo→active,tailscale ip -4inside the guest →100.88.223.70), andm93pis online despite a stale peer record (Online: falsein the JSON, buttailscale ping→ pong in 1ms and SSH over the tailnet returnedm93p / up 11 days) — the second occurrence of that pattern after 2026-10-26. The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged at 54 (53 live IPv4 peers +hermes); JSON reports 76 peers — 53 with a100.IPv4, 23 IPv6-only funnel ingress. No missing peers, no IP mismatches, CT601 still absent from both hypervisors. Container identity re-confirmed from inside all 34 guests viapct exec <vmid> -- tailscale ip -4on both hypervisors; the 3 stopped guests (CT209/CT255/CT555) and the 3 running-but-unconnected guests (CT105/CT1264/CT100) confirmed. - Updated incidents/2026-10-01-cliproxy-connection-auth-expired — the residual claim "the connection credential itself was never re-authenticated" is falsified. A live
POST /v1/chat/completionsforcliprox/terraagainstcustom:Omniroute:20128(CT600) returns HTTP 200 — the connection has since been re-authenticated, independently of the incident's remediation. The same pin oncustom:omni-101(CT101) still returns HTTP 401 (No active credentials for provider: cliprox.), so the two nodes carry separate credentials. Claim marked superseded rather than deleted (SCHEMA rule 9); the remediation itself (moving 3 jobs off the pin) was never reversed and remains correct. - Updated incidents/2026-10-05-dead-model-pin-repin — every claim re-probed live: all 7 jobs are still on
custom:omni-101with routing aliases (free-stack×6,smart-route×1) ✔; both aliases return HTTP 200 ✔;ollama-cloud/qwen3.5:397bstill HTTP 410 (retired 2026-09-25) ✔;nvidia/deepseek-v4-flashstill HTTP 400 (absent from the live catalog) ✔;command-code/gpt-5.6-terranow returns HTTP 401 (credential) rather than the original 403 (plan-gated) — the pin is still dead, only the failure signature changed. Frontmatterupdated/verified_onbumped to 2026-10-29 withcreatedleft at 2026-10-06 as the data period. - Updated systems/scheduled-tasks — the Known-Issues entry repeating the "never re-authenticated" claim now carries the correction. The page's fleet figures were re-checked against live
jobs.jsonand remain accurate: 29 jobs / 26 enabled / 3 disabled, 5 non-cleanlast_status. - Verified, no change needed: systems/homepage-dashboard (CT201
ss -ltnshows 8084 and 8085 listening, no 8082 listener, Homarr on 8102 — all as the page claims); CT201 Serve routes (/→ 8085,/mgr→ 8084 on both:443and:8443, exactly as documented, Steel routes still gone); CT460 Serve routes (12 paths on:443+:8443/webhookFunnel +:9443, matching the page). - Not changed, flagged for a human: the vault mirror at
/home/hermes/wiki/agent-estate-wiki/has drifted on both conventions — top-level 35 match / 3 stale / 72 missing,wiki/41 match / 53 stale / 16 missing. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed. - Removed/archived: none. Secrets written: none (credential names and HTTP status codes only — no key or token values).
- Unresolved:
cliprox/terraremains 401 on CT101 — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node.
[2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
- New page — incidents/2026-10-29-conduit-voice-silence: every Conduit voice call connected and then died
silently. Three faults sat in series, each one masking the next: (A)
voice.speech_server.urlwas the truncatedwss://…:8443/v1/, which the plugin dials verbatim, so the Mac answered 403 and the plugin blamed the key; (B)dashboard.basic_auth.session_ttl_secondswas the 12 h default, so an expired phone session got WS 4401 with no Hermes turn and no error on the device (the iPhone re-authenticated at 08:00:59, minutes after a silent call); (C) both host providers were dead —tts.provider: openai→ Mac/v1/speech502,stt.provider: groq→ 403/Cloudflare 1010 (blocked, not mis-keyed). Fixed 08:02–08:4x: URL →…:8443/v1/realtime, TTL → 2 592 000 s (30 d),stt.provider: local(faster-whisperbase),tts.provider: ollama_local; dashboard restarted 08:03:26. Verified by driving the whole call path for real, then by two live calls at 08:38:04 and 08:49:36 — the first entries ever inconduit-calls.json → history, so the memory fix is no longer evidence-free. - Updated systems/conduit — new runtime-checks section, the two real calls on record, the incident cross-reference,
and
verified_on: 2026-10-30. - Updated systems/hermes-agent — new dated section 2026-10-30 00:0x AWST for the voice path (fault table +
why it stayed invisible + the two-TTL-keys warning + backups), and the
tts.providerblock now carries a second superseded note recording the live values (ollama_local/local,/v1/realtime). - Updated systems/scheduled-tasks — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
superseded, stage 1's delivery column corrected to
local(verified live: stage 1 no longer delivers), both radar rows annotated with their 2026-10-29 result, and a new Resolved 2026-10-29 block under Known Issues. - Updated runbooks/job-radar — new dated slot section 2026-10-29 07:00 (handoff written → consumed →
delivered: 170 discovered / 33 duplicates avoided / 37 verified; 20 pages + 2 cover letters,
missing_cover_letter_count 0, Notion board 455 → 475), plus the note that the second real cycle completed. - Updated incidents/2026-10-27-job-radar-budget-exhausted and
incidents/2026-10-28-job-radar-stage1-iteration-cap — both resolved. Stage 1 ran 07:00:59 → 10:16:12
(3 h 15 m 13 s), closed
completed, and both ledger rows it had opened (dca848_e93a538ade6b,dca848_0a2f03b5952b) closedresolvedat 10:16:12. Stage 2 consumed the handoff at 10:40:59, closedcompletedat 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed. - Updated current-state, index — new Recent Changes block and the refreshed page descriptors.
- Cron health: window 2026-10-29 settled 176
completed/ 0failed— the first zero-failure day since 2026-10-24 — with 10 deliveries, all 10delivered. Fleet unchanged at 29 jobs (26 enabled / 3 disabled); non-cleanlast_status5 → 3 (all three are stale re-runs of long-standing rows) and everyfailure_streakreads 0. Unclosed incident rows 4 → 1, the lone survivor being the Facebook monthly's0e753e_e7f8194a7d06, which cannot clear before its 2026-11-01 slot. Pins unchanged. - Vault mirror catch-up: the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
audit were mirrored into
/home/hermes/wiki/agent-estate-wiki/, includingsystems/conduit.md,runbooks/job-radar.md,systems/scheduled-tasks.md,systems/hermes-agent.md,current-state.md,index.mdandlog.md. The wiki-maintained pages underinfrastructure/remain a separate, known gap. - Removed/archived: none. Secrets written: none — provider names, key names and HTTP status codes only, never a key or token value.
2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state
- Ran
wiki-maintain.sh. Regeneratedsystems/skills-index.md(302 skills / 69 categories — unchanged; the generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wroteraw/stocktake/freshness-20261030.txt(0 stale / 109 fresh / 3 no-date of 112 pages), rebuilt Quartz (654 files) and deployed topublic/. - Freshness is not accuracy. The age scan found nothing, but a live-vs-page diff against the running estate found real drift on five pages. All corrections below were verified against live sources before being written.
- infrastructure/tailscale — node table rebuilt.
tailscale status --jsonnow reports 50 non-funnel nodes (49 live IPv4 peers +hermes), not the 54 the page claimed. 2 IP changes (anthonys-mac-mini100.74.17.15 → 100.76.60.37;omarchymac100.84.180.69 → 100.126.173.115 — the peer formerly listed asomarchymac-1) and 4 nodes dropped out of the tailnet entirely (anthonys-imac,cachyos,omarchymac-1,omni). Every offline age refreshed.m93preadsOnline: falsefor the third consecutive audit yet is plainly up (tailscale ping→ pong, SSH →up 12 days), so the row records it online. Total peers 72 (49 IPv4 + 23 IPv6-only funnel). CT460's:443route set matched the page exactly; CT201 serves only/→ 8085 and/mgr→ 8084 on both:443and:8443. - systems/scheduled-tasks — fleet ledger corrected. Live
jobs.jsonshows 29 jobs (25 enabled / 4 disabled), not the 28/25/3 on the page:ea79a2107e36(Mnemosyne Health Monitor) was auto-paused 2026-10-30 01:04:22 after its 00:01:08 slot closedunknownon a gateway-connection failure. Non-cleanlast_statusis 4 (not 3), and the standing "everyfailure_streakreads 0" claim was false (ea79a2107e36= 1). "Failing (delivery only)" is 2, not 1 (fad5a86979d4joined066bb5191280). The retired DHL row was dropped from the job table (it lives in Removed Jobs). Seven job rows carried staleerrorclaims for jobs that have since cleared took— each was marked superseded with a dated correction rather than rewritten (SCHEMA rule 9). Theexecutions.dbretention floor note was corrected to the live 2026-10-24 09:22:36. Window tallies: 2026-10-29 = 177 / 0 and 2026-10-28 = 169 / 2 (both +1 on the page — the 23:59 Wiki Audit row had still beenrunningat audit time). - New incident page: incidents/2026-10-30-mnemosyne-health-monitor-paused. The Mnemosyne Health Monitor's own
failure notice failed to send (
send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed) —deliveries.dbholds zero rows for 2026-10-30, so the incident row is the only trace. The job was not re-enabled and the gateway was not restarted by this run; root cause is unverified and recorded as such. - infrastructure/hosts — four corrections. CT201's LAN address is
192.168.178.77, not192.168.178.166; CT450's Hermes Agent is v0.22.5 (2026.9.24), localbf0203a794a0— the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both (a stale root-level/usr/local/bin/hermeson CT450 still reports v0.19.0 and is not the live agent — the live one is thehermes-user install under~/.hermes/hermes-agent); the cron count is now 29/25/4; and the orphanedomnipeer has expired out of the tailnet entirely (no peer namedomni, no peer at 100.93.204.4), so the "offline 69d" description no longer applied. The skills count of 302 was left alone — it is correct. - systems/homepage-dashboard — five corrections, and a second pass to fix my own first pass.
"All Apps" is 114 → 123 live apps, not 122 (the API dedupes by
(name, normalised href)across groups; the literalAll Appsgroup inservices.yamlholds 122 and is not what the tab bar counts). The grid is a responsiverepeat(auto-fill, minmax(230px, 1fr))(served/dash.cssline 3640), not a fixed 5-column grid. The tab list was wrong — but my first correction was wrong too and had to be redone: the live tab bar is built from the service groups (renderTabs()iteratesstate.groups), not from thelayout:block insettings.yaml. That block and itstab:/columns:keys are inert —grep -c "Overview" /dash.js= 0 andgrep -c "columns" /dash.js= 0. There is no "Overview" tab; the live tabs are☆ Pinned,All (123)and the 14 groups (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20, Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved theme is{"pair":"Boarding Pass","mode":"light"}— one of 59 theme pairs; its light-mode amber is#795000, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts.manager.pyruns inside thehomepage-managercontainer (its containerCmdis thepip install … && python3 /opt/homepage/manager.pyline, confirmed viadocker inspectanddocker top); onlyasset_proxy.pyis a host process (0.0.0.0:8085 users:(("python3",pid=4038))). And the "no Traefik container exists" evidence was wrong — theruntipi-reverse-proxyTraefik container exists but is stopped (Exited (255) 2 months ago); the conclusion that Traefik does not occupy 443 stands. The same evidence error was corrected in infrastructure/tailscale. - current-state and index updated with the new incident and the corrected fleet numbers.
- Verified, not assumed: CT600 running and CT601 absent from both hypervisors (
pct config 601→ no such config onpveorm93p); the CT201:443/:8443Serve tables; CT201 listeners 5173/3000/8082 (none — Steel's removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row (read fromjobs.json,executions.db— whosecron_incidentstable lives inside it, the standalonecron_incidents.db/incidents.dbfiles are empty stubs — anddeliveries.db). - Not changed:
systems/skills-index.mdcount (302 is correct), CT460's Serve route table (matched live exactly), and the steel-removal records across the wiki (confirmed still true). - Secrets written: none — provider names, key names and HTTP status codes only, never a key or token value.
[2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
Verdict: one large undocumented event found and written up, and one documented incident found to have been recorded wrong.
- NEW PAGE — incidents/2026-10-30-ct460-host-outage-scheduler-gap. CT460 was dark for 5 h 30 m on
2026-10-30 and no monitor alerted. Evidence: zero
executions.dbrows between 10:58:20 and 16:37:21; PVE hostpveboot time 2026-10-30 16:24:00; CT460 boot 16:28:00; the previous gateway life (pid=3913841, started 2026-10-28T17:21:58Z) logged UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death"); scheduler resumed 16:37:21 with an 8-run catch-up burst through 16:38:22. The day still closed 138 executions / 0 failed against 10-29's 177 / 0 — ≈39 lost slots, no failures. Root cause not established: no OOM evidence locally or on the PVE host, so the ledger string is read as a candidate list, not a diagnosis. Still open: the two jobs that could have detected it (cbfe434a3b5eNFS mount monitor,1a8e90ea2271m93p LXC boot monitor) are disabled. - CORRECTED — incidents/2026-10-30-mnemosyne-health-monitor-paused was right about the record and wrong
about the run.
ea79a2107e36's 2026-10-30 00:01:08 slot was still running at the 03:3x reading; it closedcompletedat 10:02:48 after 10 h 01 m 40 s and delivered, auto-closing incidentea79a2_d93c3acdc4f2resolvedat 2026-10-30T10:02:48.662+08:00. Live job state is nowenabled: true,last_status: ok,failure_streak: 0. The page's "zerodeliveries.dbrows / the notice was never delivered" claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with 8 delivery rows, alldelivered(latest 18:46:04). Page frontmatter movedactive→resolved, with the old snapshot kept below as explicitly historical. ⚠ Its 12:01 slot was lost to the host outage and is the still-running16:37:21 catch-up. Reconciliation lesson: a slot mid-flight is not a failure — checkfinished_at, notlast_status, before writing an incident. - Reconciled —
systems/scheduled-tasks.md. Fleet 29 jobs (26 enabled / 3 disabled) — the "4th disabled job" reading is superseded (frontmatter date, fleet block, Total jobs, non-cleanlast_status4 → 3, failing 1 → 0, the health-monitor table row, the disabled table row, and a new "Resolved 2026-10-30" subsection). Non-clean now 3 = 1error+ 2delivery_failed; unclosed incident rows 2 → 1 (survivor: the Facebook monthly's0e753e_e7f8194a7d06, due 11-01). - Reconciled —
current-state.md(new leading cron entry + superseded marker),infrastructure/hosts.md(cron row),index.md(frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),incidents/2026-10-30-mnemosyne-health-monitor-paused.md. - Checked, unchanged: 12 sessions active in-window (longest
local-40472e0b…, 1,351 messages), no orphanedrunningrows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written.
[2026-11-01] audit | Nightly review — Conduit 0.122.2, the Local Live fault, a force-drained reload, dead Asana paths
- NEW PAGE — incidents/2026-10-31-conduit-voice-local-mode-not-applying. The app's "local" voice mode
never applies even though its connection test succeeds — because the test checks socket reachability only.
Every local turn is still impossible: the Mac mini speech server's LLM proxy is disabled (LLM endpoint →
HTTP 501, "start the server with
--enable_llm_proxy"),/v1/usagereadsllm_proxy.requests: 0across 25 observed turns, and its realtime socket accepts the connection but rejectssession.updateas an unknown event.voice.modeis a phone-side selector: nomodekey exists inconfig.yaml → voiceorconduit-settings.json(whose only device state isdevices/<UUID>/mode: small×3), so "the change did not apply" cannot be diagnosed on host config. Evidence from session20261031_193709_40b9c2(started 19:37:09, 30 msgs / 17 tool calls, still open at the 00:2x reading);voice.local_live.front_modelwas repointed tomistral/voxtral-small-latestmid-session (config.yamlmtime 21:02:30). Nothing was changed on the Mac. ⚠ One misread was caught in-session and is recorded as a lesson:local_live.py:882is the only writer of_local_live_last_request, inside the chat-completions route — so a "recentlast_request_at" was the investigation's own probe, not phone traffic (last_request_atisnull). - NEW PAGE — incidents/2026-10-31-gateway-reload-force-drain-orphaned-run. The 18:06:41 gateway reload
waited the full 1,800 s drain cap and force-drained at 18:37:38 (new gateway 18:38:43), orphaning
ea79a2107e36(Mnemosyne Health Monitor, 12:01:06 slot*, 6 h 38 m 09 s in) which closedfailedat 18:39:15 on "lost its durable fire claim ownership". Derivative watchdog reds at 18:39:23 / 18:45:23 / 19:00:22 cleared 19:15:21. A second, clean restart followed at 19:23:39 → 19:24:40. Same class as the 10-26 planned-restart orphan — second instance, first on a plain reload. - NEW PAGE — incidents/2026-10-31-asana-access-failures. Both Asana paths on CT460 are non-functional:
the MCP server
asana(https://mcp.asana.com/v2/mcp,auth: oauth,enabled: true) has never authenticated (~/.hermes/mcp-tokens/holds onlyasana.client.json, 319 B, mtime 2026-10-26 01:07; logs read "MCP OAuth setup failed for 'asana' … no cached tokens found. Runhermes mcp login asanainteractively first"), and the hosted connector rejects the call shape. The repair needs an interactive login, which cron cannot perform. - UPDATED —
systems/conduit.md(13,184 → 18,722 B; frontmatterupdated/verified_on→ 2026-11-01). Version0.69.2/942ddb8→0.122.2/9eac857(10-26 reading kept explicitly superseded); the ⚠ "uncommitted patch, five releases behind" warning is RESOLVED — the tree is clean andMEMORY_TIMEOUT_S = 10.0ships upstream indashboard/api/memory.py(with await_forbackstop andtests/test_memory.py), so the 10-23 in-tree edit is no longer needed; dashboard gatewaypid 4042003(10-29 08:03:26) →pid 176771(10-31 19:35:02); the call-store shape change is recorded (v: 1;historyempty;last_request_at: null;ends= 2 ×outcome: stoppedat 21:06:05 / 21:10:48;settings.quiet_at_screen: true= #479); data volume 930.0 MB → 938.2 MB (983,793,664 B) with 2,623,879events/ 4,384buckets/ 504windows(≈8 MB/day, no retention policy); new dated section;Open items— patch item marked resolved, three new local-live items added. - UPDATED —
systems/scheduled-tasks.md(same pass): fleet 29 jobs (26 enabled / 3 disabled); 2026-10-31 closed 160 / 4; non-cleanlast_status4 (2error+ 2delivery_failed); failing jobs 1 (ea79a2107e36,failure_streak2); superseded readings marked rather than overwritten. - UPDATED —
systems/hermes-agent.md: Asana MCP auth-state re-verification note (enabled but never authenticated) + both 2026-10-31 gateway restarts (18:06:41 force-drain, 19:23:39 → 19:24:40 clean). - UPDATED —
index.md: frontmatter and banner → 2026-11-01; Conduit entry (version, patch resolved, gateway pid, new fault, empty call history); three new incident entries added above the 10-30 pair. - UPDATED —
log.md: this entry. - Checked, unchanged: ship rows before 10-30 (nothing new);
inc_20261031_000001(gateway-update) is still open,inc_20261030_000001closed 2026-10-30 18:05; 3deliveries.dbrows for 2026-10-31 carry a failure error (child run failed: exit status 1) — they are the failing runs' own notices, not new faults; no page archived; no secrets, tokens, passwords or keys written. - Probes (context, not findings):
omniroute-1100.78.63.20:20128→ HTTP 200;m93p100.104.51.98andpve100.96.100.82answered neither ICMP, SSH nor:8006from CT460 at the 00:2x reading — recorded as unverified reachability from this container, not as a host outage. - Mirror catch-up (a whole-tree pass, beyond this audit's own pages). Because the previous copy bridge only
ever synced the pages an audit touched, the user-facing vault had silently drifted. This pass re-copied every
committed page in the authoring tree into both vault layouts (
/home/hermes/wiki/agent-estate-wiki/and.../wiki/) — 61 pages — which closed the 9-page drift and picked up everything the 2026-10-31 audit had changed and never mirrored (current-state.md,incidents/2026-10-30-ct460-host-outage-scheduler-gap.md,incidents/2026-10-30-mnemosyne-health-monitor-paused.md,infrastructure/hosts.md,systems/mnemosyne.md) plus 53 older stragglers. Deliberately left unsynced: the 6 files the 2026-10-31 03:00 stocktake has left uncommitted in the authoring repo (infrastructure/docker-services.md,infrastructure/proxmox.md,infrastructure/tailscale.md,systems/messaging-integrations.md,systems/omniroute.md,systems/skills-index.md) and untrackedraw/stocktake/freshness-20261031.txt— the mirror convention is "(authoring COMMIT_SHA)", so uncommitted content stays out until the stocktake's own commit lands.
[2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago
- BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake).
wiki-maintain.shexited 1 at step 3/3 with an esbuild panic. Root cause was not esbuild:incidents/2026-10-31-conduit-voice-local-mode-not-applying.mdhad an unquoted YAMLtitle:containing the substring:(from"voice mode: local"), so YAML parsed it as a mapping entry and Quartz aborted withbad indentation of a mapping entry (2:27). A scan of all 117 pages found exactly one offending frontmatter line; the title is now double-quoted with escaped inner quotes. This is a latent trap worth remembering: any future page title containing": "or":unquoted will silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather than a YAML error. - STALE PAGE —
incidents/2026-10-09-memory-recall-relevance.md(23d) — RESOLVED, not stale. The page's central claim was that the recall-ranking fix "was reported saved and is not present in live config", and that conflict152was "the only open conflict". Both claims are false, and the page was the reason the incident looked open for three weeks. Live~/.hermes/mnemosyne/config.yamlreadsvec_weight: 0.5,fts_weight: 0.3,importance_weight: 0.3(plusMNEMOSYNE_IMPORTANCE_WEIGHT=0.3in the environment) — the weights live in Mnemosyne's own config file, not in~/.hermes/config.yaml'smemory.mnemosyneblock that the 2026-10-10 probe read. The 2026-10-10 verification probed the wrong file and concluded "not applied". The original regression test now passes:mnemosyne recall "Are you up to date"returns only substantive current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode (id 32f469aae09d56dc, still in the DB) no longer outranks anything. Conflict152was auto-resolved 2026-10-29 23:53:47Z ("low-confidence LLM extraction noise") — it has not been open since. All three open items closed; page moved tostatus: resolved. Live tier counts at 2026-11-01: working 941, episodic 1,124, canonical 90, conflicts 175 with 11 open (ids 165–175, all one-batch extraction noise sharing a singlefact_a_id), working embeddings 927/941 (98.5 %), episodic 1,124/1,124 (100 %) — both above the 95 % floor, noreindexneeded. - UPDATED —
systems/mnemosyne.md: stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01 block added, including the weights' actual home and the resolved-incident pointer. - UPDATED —
infrastructure/tailscale.md: table rebuilt from livetailscale status --json— 1 state transition and 5 offline-age drifts, no IP changes, no missing peers.omarchymac(100.126.173.115) went offline (Online: false,LastSeen 2026-10-31 13:20Z,tailscale pingtimes out — a genuine transition; the JSONOSfield readslinux).m93p's peer record readsOnline: falseagain — the 4th consecutive stale record — whiletailscale ping→pongand SSH returnedup 1 day, 12:02, so the row stays online. Ages refreshed: cloudreve 98→99d, freshrss-new 74→75d, omarchybook 9→10d, openmausbot 9→10d, ubuntu-server-xfce 71→72d;conduit(55d) andelementsynapse(102d) were already correct. - UPDATED —
infrastructure/docker-services.md: CT201's Tailscale IP was wrong — the page read100.116.116.116; livepct exec 201 -- tailscale ip -4returns100.96.244.39(which[[infrastructure/hosts]]already had right). The CT200 inventory had three misspelledovertchat-*entries (writtenoverchat-*) and one stale entry,worldmonitor-seeder-fast, which is not running — the list is now regenerated verbatim from livedocker ps. CT333's first container was renamed:adoring_jemison→thirsty_payne. Counts re-verified: CT200 61, CT201 20, CT333 8, CT1001 1 (sshwifty), CT460ignisrunning. - UPDATED —
infrastructure/proxmox.md: storage percentages refreshed against livepvesm statuson both hosts (pvefred-qnap82.45→82.77 %,SSD163.80→64.32 %,local-lvm13.60→13.75 %;m93pnfs-proxmox-*82.45→82.78 %,omb-internal58.00→57.92 %).pct list/qm liston both hosts re-confirmed — no CT601 on either hypervisor, CT255 stillrunning, VM 103cachyos-m93pstopped, VM 106openmausbot-vmrunning.m93puptime recorded as up 1 day, 12:02 with a load average of 7.66 / 12.80 / 11.32 — elevated, worth watching. - UPDATED —
systems/omniroute.md/systems/messaging-integrations.md: re-probed today — both OmniRoute nodes answer 401 on/v1/models(alive, auth required), CT600 and CT101 bothrunning; Mattermost'ssystemctl is-active→active,LISTEN *:8065,/api/v4/system/ping→ HTTP 200. Frontmatterverified_onbumped to 2026-11-01. - REGENERATED —
systems/skills-index.md: 301 unique skills in 68 categories, 46 archived. The count is unique(category, skill_name)pairs, notSKILL.mdfile count — livefindreturns 302 files, and the one difference is the known nested alias duplicateproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.mdvsproductivity/obsidian-git-sync/SKILL.md. 301 is correct — do not "fix" it to 302. One skill was archived since the last run:hermes-runtime-worker-diagnostics(46 archived, up from 45). - FRESHNESS: the 2026-11-01 report lists 1 stale / 113 fresh / 3 no-date of 117 pages. The single stale page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after the fixes, per the script → fix → re-run → commit order.
- Probes (context, not findings):
m93pSSH over the tailnet timed out on the first two attempts and succeeded on the third — a flaky path, not a host outage (tailscale ping→ pong throughout).homarron CT201:8102 returns 307 → 200 on/board, so the dashboard page's "live, 200" claim still holds. CT201's:8443/root returns 502 (upstream 8082 has no listener) and:8443/mgrreturns 200 — both still as the homepage-dashboard page records. No secrets, tokens, passwords or keys written.
[2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by -15, and a red status with no row behind it
Window: 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on state.db (sessions),
~/.hermes/cron/{jobs.json,executions.db,deliveries.db}, gateway.log / agent.log.1, and both git trees.
- NEW —
incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md(status: active): the Sunday 18:0079f9bb41ecf5closedunknownat 18:05:13 with "Gateway connection failed; no local fallback", delivered nothing, and its incident row79f9bb_9c443b602eecis stillalerted.deliveries.dbhas no row after 11:01:12. Batch-mateb63ed5817212hit the identical error and self-healed in 68 s. The same page records the inverse silent failure:ecc1fc13655areadserror,last_run 11-01T12:01:00,failure_streak: 1— with no execution row and no incident row. - NEW —
incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md(status: resolved): three workersSIGTERMed (-15) after adopting their claim (85edf6772bca07:30:46;96eb95ce12f2+0e753e9720fd09:02:18) → "whether side effects ran is unknown". Fourth variant of the orphan class, and the first with no gateway restart/reload/drain anywhere in the window (PID175269up since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a 30 s worker-ack window vs a ~70 s cold start (Plugin discovery complete: 83 found, 69 enabled). All three incident rowsresolved; one execution leftunknown. - UPDATED —
systems/scheduled-tasks.md:## Countadvanced to the 2026-11-02 window (167completed/ 1running/ 2unknownvs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the first non-clean day in four); fleet 29 jobs / 26 enabled / 3 disabled unchanged; unclosed incident rows 1 → 1 but a different row (all three rows the 10-31 audit left open closed on 11-01, including0e753e_e7f8194a7d06, open since 2026-10-01 — 31 days); new## Re-verified 2026-11-02section covering the three failure modes, the 09:00 cold-start pair and the ledger. Non-cleanlast_status4 → 2 (ea79a2107e36,79f9bb41ecf5). - UPDATED —
systems/messaging-integrations.md: Telegram degraded all day (dual-stackapi.telegram.orgConnectTimeout→ IPv4-literal fallback; disconnect 20:51:53, polling restarts 22:33:30 / 23:12:23), andgetent hostscurrently returns only an AAAA record forapi.telegram.org— consistent with, but not proof of, a systematic timeout on the path DNS hands the client. Therelayplatform has had no adapter for ~29 h (Reconnecting relayattempt 147 @10-31 18:39:24 → 336 @11-02 00:10:47, counter never reset — itself the proof of no gateway restart).066bb5191280'sthread_id 34795home-DM miss is the second confirmed variant of the thread-mismatch bug: the thread-less fallback fires, the content lands, and the run is recordedfailedanyway. Day's delivery health: 8 rows / 7delivered/ 1failed. - UPDATED —
index.md:updated/verified_on→ 2026-11-02, the audit summary line rewritten for this window, and the two new incident links added to the incident list. - Not done / unchanged: no page removed or archived; no job added, removed, disabled or repinned; no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts below instead of a bulk copy) — an audit's mandate is smaller than a full sync.
- DRIFT (context, not findings): the authoring tree was clean and in sync with Gitea
(
origin/master..HEADempty) before this audit; touched pages were mirrored into the vault's existing conventions and hash-verified. No secrets, passwords, tokens or keys written or quoted — every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
[2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see)
- Scope:
state.dbsessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST;cron/executions.db,cron/deliveries.db,cron_incidents,cron/jobs.json(all read-only). Authoring tree/home/hermes/workspace/agent-estate-wikiwas clean and in sync with Gitea (origin/master..HEADempty) before this audit;systems/skills-index.mdandraw/stocktake/freshness-20261102.txtwere committed on the stocktake's behalf (see below). - NEW —
incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md: the third iteration-cap in seven days (10-27 tool-call budget, 10-28 cap, 11-02 cap) ondca8482e4f76(07:00:33 → 16:01:52, 9 h 01 m 19 s), and the first that raised nocron_incidentsrow at all — it closedcompleted/error: NULL/delivery_outcome: suppressed. Both Stage 2 slots exited[SILENT], so nothing reached Notion, no cover letters were written and the 164 deduped candidates were never persisted. Page carries the "green row, no artefact" detection table contrasting the three occurrences, and records the mid-run URL-canonicalisation fix (au.seek.comvsau.seek.com.au) that had been silently dropping SEEK and Indeed entries from dedup. - UPDATED —
incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md: statusactive→resolved. Live evidence: the same execution row (be6ec11a8e68428084af9ab18fb148db) closedcompleted/deliveredat 01:27:22 after 7 h 26 m 43 s, delivery at 01:27:21.802246, incident79f9bb_9c443b602eecauto-closed at 01:27:21.606691. The 18:05:13unknownwas a reconciler timing artefact on a still-running job, not a verdict. Resolution section added; open questions 1–3 answered, question 4 retained (the two unreconciled admission files141448f9…json/69b5588f…json). - UPDATED —
systems/scheduled-tasks.md:## Countadvanced to the 2026-11-02 window — 177completed/ 2running/ 0unknown/ 0failed(179 rows) vs 167/1/2 on 11-01, 160/4 on 10-31, 138/0 on 10-30; fleet unchanged at 29 jobs / 26 enabled / 3 disabled; unclosed incident rows 1 → 0 (the first zero in the period); both 11-01unknownrows resolved andecc1fc13655a's ghosterrorcleared; deliveries 9 rows / 9delivered. New## Re-verified 2026-11-03section documents the two invisible failures, the stocktake artefacts committed, thesubstr(started_at,1,10)grouping rule and the "do not readrunning/unknownas a verdict before the expected runtime has elapsed" correction. Frontmatterupdated/verified_on→ 2026-11-03,review_after→ 2026-11-10. - UPDATED —
systems/browser-backend.md:updated/verified_on→ 2026-11-03 plus a## Historyentry for the 2026-11-02 REST-server wedge (requests timing out with nothing reaching the service log, cleared bysystemctl --user restart camofox-browser.service; the four-board Job Radar run then proceeded). Live re-probe 2026-11-03 00:16:14 AWST: unitactive,MainPID 953299,ActiveEnterTimestamp Mon 2026-11-02 18:11:04,MemoryCurrent 37.5 MB,/health→ 200{"ok":true,"engine":"camoufox","browserConnected":false,…},/tabs→{"running":true,"tabs":[]}. Page now states explicitly that this wedge is a different fault from the 2026-10-12 memory ceiling (hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control. - UPDATED —
index.md:updated/verified_on→ 2026-11-03, the audit summary line rewritten for this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated ✅ RESOLVED 2026-11-02T01:27:21. - COMMITTED on the stocktake's behalf:
1a8e90ea2271finished withsystems/skills-index.mdmodified andraw/stocktake/freshness-20261102.txtuntracked. Both are valid artefacts — freshness 4 stale / 113 fresh at a >21-day threshold, skills-index regenerated with Devops 108 → 109 (pulse-turn-handling) — so this audit committed them rather than let the next slot overwrite them. - Not done / unchanged: no page removed or archived; no job added, removed, disabled or repinned; no decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a bulk copy).
- DRIFT (context, not findings): touched pages follow the vault's existing conventions and SCHEMA.md frontmatter rules. No secrets, passwords, tokens or keys written or quoted — every job id, PID, thread id and timestamp above is a non-sensitive operational identifier.
[2026-11-04] audit — window 2026-11-03
Verdict: one genuine red row, and the ledger produced it itself. 2026-11-03 settled 170 completed /
1 failed / 1 running / 0 unknown (172 rows) against 11-02's 177 / 2 — the first failed row since
10-31.
New incident page: incidents/2026-11-03-job-radar-stream-active-timeout.md — Job Radar Stage 1
(dca8482e4f76) ran 07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s and died on
RuntimeError: stream active timeout. Its session (local-7180cef7…) compacted five times (08:04:35,
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went silent for 65 m 21 s after its last tool result at
12:23:28 before the harness recorded "This turn did not complete." at 13:28:49. Incident
dca848_dfd4bea84f2a (failure_type: timeout) opened 13:28:51.463929, alerted 13:28:55.141618, still
open → fleet unclosed rows 0 → 1. The failure notice was delivered at 13:28:51.741724 and Pulse
carried it to Anthony (report p34c8e899-1, desktop session 20261031_190813_95fbde: "…dying on a stream
active timeout").
Updated: incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md — new 11-03 row in the
recurrence table plus a 2026-11-04 update section that answers its open question 3: output delivery
tracks the terminal state, not the run length. 11-02 ended completed → delivered nothing; 11-03 ended
failed → notice delivered and a row raised. Open question 2 narrows: the ledger can record a non-clean
state when the harness kills the turn; the iteration cap is the case that still escapes as completed.
Question 4 is answered against us: 2026-11-02's candidates and 11-03's discovery are both unrecovered.
Second lost Stage-1 day: both Stage 2 slots declined silently again (5fc292ff0f3b 11:30:36 → 11:32:16,
14:30:32 → 14:33:11) and ~/.hermes/radar-ref/handoff/ still holds only 10-27, 10-29, 10-30.
New page: systems/hermes-hq-push.md — plugin hermes-hq-push v0.9.27 (kind: standalone, author
Hermes HQ), tree mtime 2026-11-03 11:06, enabled (hermes plugins list → enabled, state user;
config.yaml plugins.enabled line 7556). Its own manifest states it replaces dispatch-push, and the
replacement is clean: no dispatch-push reference survives in config.yaml and no such directory
remains. It is a different stack from the Cadu/Vory bridge on systems/push-notifications.md (untouched,
still degraded). Operational note recorded: the plugin's pre_tool_call guard rejects any tool call
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
blocked and had to be rephrased.
Fleet: 29 jobs — 26 enabled / 3 disabled (unchanged). Non-clean last_status 1 → 2:
dca8482e4f76 (error, streak 1) joins 066bb5191280 (unchanged delivery_failed). Deliveries: 7 rows,
7/7 delivered (07:13:49, 07:30:35, 08:39:25, 12:37:09, 13:28:51 for_failure, 13:49:11, 18:45:39).
⚠ The 03:00 stocktake ran long for the second day: 1a8e90ea2271 03:00:30.495724 → 10:41:13.851030 =
7 h 40 m 43 s, completed / suppressed, delivered nothing, and again left its artefacts
uncommitted (systems/skills-index.md modified, raw/stocktake/freshness-20261103.txt untracked) — this
audit committed them.
Sessions in the window: 16 (14 cron, 1 ios 20260908_031541_49c051, 1 desktop
20261031_190813_95fbde) — 14 started, all cron. No gateway restart: PID 175269 is still the
2026-10-31 19:23:56 start (hermes-dashboard PID 176771, 19:35:02).
Not done / unchanged: no page removed or archived; no job added, removed, disabled or repinned; no decision reversed. No secrets, passwords, tokens or keys written or quoted — a Gitea remote URL with an embedded credential was seen during verification and is not reproduced here ([REDACTED]).
[2026-11-05] audit - window 2026-11-04
- New page:
incidents/2026-11-04-upstream-stream-failure-0700-cohort.md- two 07:00 jobs (dca8482e4f76Daily Marketing Job Radar - Discovery,85edf6772bcaTech & AI Newsletter Digest (Analytical)) died 36 s apart (09:41:44 / 09:42:20) onRuntimeError: Upstream stream failed before completion.after 2 h 41 m each, both oncustom:omni-101/smart-route; both incidents still open (state = alerted). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own candidate files, re-verified every candidate live, and wrote 5 roles (board 479 -> 484) + 1 canonical cover letter (MISSING_COVER_LETTER_COUNT 0), consuming the handoff at 16:08:59. - Updated
incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md- CT299 was frozen twice, not once. Second freeze: a manualvzdump 299 ... --storage ssd0started 2026-11-04 08:30:04 and wedged (unexpected status) until 11-05 ~00:30. Also records the earlier clean failure at 07:36:44 (logvzdump-lxc-299-2026_11_04-07_36_44.logon the dump target) and threepush_filetasks from CT460 at 07:17-07:33, plus the abandoned/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299. Live probe 2026-11-05 00:54:21 AWST:LOCK: 0, bothcgroup.freezeflags0, guestALIVE,sftpgo: active, 0 vzdump procs. - Updated infrastructure/tailscale - the sftpgo row now records recovery from two freezes.
- Updated systems/scheduled-tasks - new ledger bullet: 11-04 settled 170 rows - 167
completed/ 2failed/ 1running; non-cleanlast_status2 -> 3; unclosed incident rows 1 -> 3; deliveries 8/8delivered; the two failures are one upstream signature; stocktake overran for the third day (5 h 41 m 46 s,free-stackpin). - Updated runbooks/job-radar - cover-letter model route:
regenerate_cover_letter.pydefaults toCL_MODEL=deepseek-paid-smart(line 82); the 11-04 run pinnedgemini/gemini-3.5-flashafter HTTP 402.grepshows noCL_MODELin~/.hermes/.env- the override was process-local and did not persist, so the next run inherits the default again. Endpoint probe 2026-11-05:omniroute:20128/v1/models-> 401 and100.78.63.20:20128/v1/models-> 401 (live, key-gated) - so the route is reachable; the same model serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested. - Updated current-state - new "Recent Changes (2026-11-05)" block; the page had been sitting at the 2026-10-30 block across five audits.
- Updated index - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved 2026-11-03 -> 2026-11-05.
- Live verification this audit: pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup
freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from
CT460 on
100.88.223.70:8080;jobs.json,executions.db,cron_incidents,deliveries.dbread live. - Not done / unchanged: no page removed or archived; no job added, removed, disabled or re-pinned; no decision reversed. No secrets, passwords, tokens or keys written or quoted.
- network-map.md re-verified (live probe 2026-11-05): corrected the ct460 port table — 9377 is Camofox browser REST (
/home/hermes/services/camofox-browser), not Stratos;8790and5174are no longer listening; added9120(wiki serve),9222(Camofox CDP/WS),7777(OpenChamber),6767(Paseo),8085(AgentMail webhook),9900(gateway). Added the full Tailscale Serve table (12 routes on:443,/on:9443→ 5173, Funnel:8443/webhook→ 8085). Two routes are dead:/agent-estate→ 8769 and/synergy-prep→ 9123 have no listener.
[2026-11-05] correction + live probe | fleet count, 11-04 ledger composition, deepseek-paid-smart dispatch
- What was wrong. The 2026-11-05 nightly-audit entry (written 00:5x AWST) recorded the fleet as 158 jobs
(144 enabled / 14 disabled) and the 11-04 ledger as closing 1
item_suppressed. Both were wrong. Live re-reads -~/.hermes/cron/jobs.json,hermes cron list, and the in-flight job count - each give 29 jobs / 26 enabled / 3 disabled;executions.dbshows 11-04 at 170 rows - 167completed/ 2failed/ 1running(this audit), with noitem_suppressedstatus anywhere on 11-04. - Where corrected:
systems/scheduled-tasks.md(Count bullet + correction clause),current-state.md(11-05 block),index.md(11-05 ledger paragraph), and this entry. No other page carried the bad number. - 11-03 re-settle:
171 completed / 1 failed(172 rows) - itsrunningrow has since completed - so 11-03 and 11-04 are the only consecutive days with afailedrow since 10-31. - Live probe - model dispatch, 2026-11-05 01:33:12 AWST (first-hand, keyed):
POST http://omniroute:20128/v1/chat/completionswithdeepseek-paid-smart-> HTTP 503ALL_TARGETS_SKIPPED(poolSize: 5,attempted: 0). The same request tohttp://100.78.63.20:20128/v1/chat/completions(omniroute-1) -> HTTP 200,model: deepseek/deepseek-v4-flash. Unkeyed/v1/modelson both hosts returns 401 - auth-gating, not health. The fleet's onlydeepseek-paid-smartpin is this audit job (cbfe434a3b5e, providercustom:omni-101, i.e. theomniroute-1node), which is why the audit still completes while a direct CT600 probe 503s. - Re-verified and matching (no change needed):
cron_incidents- 3 unclosed rows, allstate = alerted;deliveries.db- 8 rows on 11-04, 8/8delivered, including bothfor_failurenotices (09:41:43, 09:42:20); non-cleanlast_status3; pin distribution 14smart-route/ 6free-stack/ 8 unpinned / 1deepseek-paid-smart. - Files touched this pass:
systems/scheduled-tasks.md,current-state.md,runbooks/job-radar.md(CL_MODELcaveat now carries the keyed probe),index.md,log.md. Frontmatterupdated/verified_onbumped to 2026-11-05 where stale. - Not done / unchanged: no job added, removed, disabled or re-pinned; no page removed or archived; no
decision reversed. No secrets, passwords, tokens or keys written or quoted - the API key was read from
~/.hermes/.envinto the shell for the probe only.
[2026-11-05] stocktake | Skills-index generator divergence FIXED at the root; Docker + Tailscale drift corrected against live probes
- Ran
wiki-maintain.sh(repo canonical path) - exit 0. Regeneratedsystems/skills-index.md(302 skills / 68 categories,.archive/excluded, 46 archived), wroteraw/stocktake/freshness-20261105.txt(0 stale / 122 fresh / 3 no-date of 125 pages), rebuilt Quartz, deployed topublic/, restartedhermes-wiki-serve.service. - ROOT CAUSE FOUND AND FIXED - the
systems/skills-index.mdflip-flop had TWO WRITERS. The repowiki-maintain.sh(mtime 2026-11-03) and~/.hermes/scripts/wiki-maintain.sh(mtime 2026-08-17) are different scripts, each with its own generator: the repo one scans withfindexcluding*/.archive/*(-> 302 skills / 68 categories); the stale one usesPath.rglob('SKILL.md')and a 2-segment category heuristic (-> 349 skills / 26 categories, because it counts the parked archive).wiki-maintenance/SKILL.mdline 569 pointed the 23:59 nightly audit at the stale path while the 03:00 stocktake ran the repo path - so each job overwrote the other's page and the wiki oscillated nightly with no error raised by either job.- Fix:
~/.hermes/scripts/wiki-maintain.shis now a delegating wrapper with no generator logic- it
execs the repo script. The repo script gained--no-buildso the wrapper honours the interface its callers were written against without carrying a second copy of the generator.
- it
- Verified by execution, not inspection:
bash ~/.hermes/scripts/wiki-maintain.sh --no-build->Written ... with 302 skills in 68 categories; both scripts passbash -n. - SKILL.md repointed at the repo path, with a note that a generator must live in exactly one place. -> incidents/2026-11-05-skills-index-generator-divergence; the "still open" item on incidents/2026-08-15-skills-index-drift is closed.
- Fix:
infrastructure/docker-services.mdcorrected against livedocker ps(all hosts, 2026-11-05 05:2x):- CT200 61 running / 68 total (was "62"); the
worldmonitor-seeder-scheduledentry recorded on 2026-11-04 as "unhealthy" does not exist on any host -docker ps -a --filter name=seederacross CT200/201/333/460 returns zero matches. Live CT200 shows exactly three worldmonitor containers (worldmonitorUp 5 dayshealthy,worldmonitor-redis,worldmonitor-redis-rest). There is no seeder container and no worldmonitor-seeder cron job anywhere in the estate. - CT333 first container renamed again:
nostalgic_leavitt->flamboyant_mahavira(imageghcr.io/github/github-mcp-server, created 2026-11-04T10:53:53Z). CT333 = 8 running / 9 total. - CT201 20 running / 26 total; CT460 1 running / 1 total (
ignisUp 5 days); CT1001 1 running / 9 total (sshwifty). Per-host "Last verified 2026-11-05" added.
- CT200 61 running / 68 total (was "62"); the
infrastructure/tailscale.mdre-verified against livetailscale status --json: the node table is exact - 52 rows, 52 live IPv4 peers, no live peer missing from the table, no table row without a live peer, and every Online/offline state matches (checked programmatically, not by eye). Two offline ages corrected:conduit59d -> 60d,ubuntu-server-xfce75d -> 76d.tailscale serve statusstill matches the recorded table exactly (12 paths on :443, plus the:8443/webhookFunnel and the:9443tailnet route).infrastructure/hosts.md: skills figure corrected to 302 skills / 68 categories (was "69 categories");updated/verified_on-> 2026-11-05.index.md: skills-index bullet refreshed to the 2026-11-05 figures (302 / 68 categories, freshness 0 stale / 122 fresh / 3 no-date of 125), the_archive/typo corrected to.archive/, and the new incident linked.- Freshness: the 8 pages stale on 2026-11-04 were all re-verified and refreshed, so the 2026-11-05 report
reads 0 stale. The 3 pages with no
updateddate areSCHEMA.md,log.mdandexperiments/active/example.md-log.mdis append-only by design, the other two are templates. - Files touched this pass:
systems/skills-index.md,infrastructure/docker-services.md,infrastructure/tailscale.md,infrastructure/hosts.md,index.md,incidents/2026-11-05-skills-index-generator-divergence.md(new),incidents/2026-08-15-skills-index-drift.md,wiki-maintain.sh,log.md. - Not done / unchanged: no job added, removed, disabled or re-pinned; no page removed or archived; no decision reversed. No secrets, passwords, tokens or keys written or quoted.
[2026-11-06] audit - window 2026-11-05
- Window reviewed: 2026-11-05 00:00-24:00 AWST; live reads 2026-11-06 00:1x-01:0x AWST. Fleet unchanged at 29 jobs (26 enabled / 3 disabled) - a fourth consecutive audit at the same count, no job added, retired, disabled or re-pinned.
- New page: incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts (status: OPEN). Three live
.envvalues (TELEGRAM_BOT_TOKEN,MATTERMOST_TOKEN,HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) sit in plaintext in 17 rows of~/.hermes/mnemosyne/data/mnemosyne.db-memories.content4,episodic_memory.content3,working_memory.content1,gists.text4,memoria_facts.context_snippet1, FTS mirrors 4 - andCAMOFOX_API_KEYsits verbatim in thepromptof two enabled Facebook Birthday jobs (0e753e9720fd,fad5a86979d4) and hence incron/deliveries.db. Method: exact-substring match of each.envvalue across every TEXT/BLOB column of every table inmnemosyne.dbplus its FTS mirrors; byte-membership scans of the 10.dbfiles under~/.hermes/config-backup/(six are 0-byte; onlycron/deliveries.dbcarries any tested value), the shared-surface DB, the seven uncompressed rotation snapshots under~/.hermes/mnemosyne/backups/(all seven reproduce the DB in full) and a chunked scan of the 1.7 GB session storestate.db, which carries all four values and is the likely capture origin (capture writes verbatim session text). Nothing was rotated, deleted or scrubbed; no credential value was printed, logged or written to any page. - Both 11-04 casualties recovered in full.
85edf6772bca(Tech & AI Newsletter Digest) completed its 07:00 run in 7 m 01 s (07:00:42 -> 07:07:43) anddca8482e4f76(Daily Marketing Job Radar - Discovery) in 5 h 35 m 35 s (07:00:42 -> 12:36:18); neither re-raisedRuntimeError: Upstream stream failed before completion.The 11-05 ledger settled 175 rows / 18 jobs - 173completed/ 2running/ 0failed, the first zero-failedday since 2026-11-02; the tworunningrows areea79a2107e36(Mnemosyne Health Monitor, 12:00:45 slot, still in flight 12 h+ with no delivery) and this audit (cbfe434a3b5e). Deliveries 10 rows / 10delivered/ 0for_failure. Fleet non-cleanlast_status3 -> 1 (only066bb5191280, Uber Eats Monthly Spend Report,delivery_failed, streak 0). - Incident ledger emptied. Unclosed rows 3 -> 0: the digest row closed 07:07:42 and both Job Radar rows
(
dca848_dfd4bea84f2atimeout,dca848_3b1bb1e9b003unknown) at 12:36:17.incidents/2026-11-03-job-radar-stream-active-timeout.md->status: resolvedwith a Resolution section;incidents/2026-11-04-upstream-stream-failure-0700-cohort.mdclosed out - the first empty ledger since 2026-11-02. - Updated systems/scheduled-tasks - frontmatter (
updated/verified_on-> 2026-11-06,review_after2026-11-07,live-sessions-dbadded tosources), a new## Countbullet for the 2026-11-06 read and a new## Re-verified 2026-11-06span (recoveries, zero-failedledger, delivery ledger, the secrets finding). - Updated systems/mnemosyne - frontmatter dates -> 2026-11-06 (
review_after2026-11-13),sources+live-secrets-scan, live-state-db, and a security entry at the head of## Known Issuesrecording that the store's own tables carry 17 rows of live credential material, that recall is the exposure path, and that only rotation covers the copies already taken. - Updated current-state, index (
## Incidentslink for the new page, header summary rewritten to the 2026-11-06 read) and this log. - Freshness:
raw/stocktake/freshness-20261106.txt(re-run after this audit's writes) -> 0 stale (>=90d), 124 fresh, 3 no-date of 127 pages, the new incident page included. The three no-date pages are unchanged (SCHEMA.md,experiments/active/example.md,log.md). - Not done / unchanged: no page removed or archived; no decision reversed;
066bb5191280and thedeepseek-paid-smart-on-omniroutedispatch fault carried, not re-diagnosed. No secrets, passwords, tokens or keys written or quoted - the incident page records where secrets live, never their values.
2026-11-06 03:0x AWST - stocktake (wiki-stocktake-maintenance)
-
Ran
wiki-maintain.sh(repo script, canonical path).systems/skills-index.mdregenerated -> 302 skills / 68 categories (unchanged count; date rolled to 2026-11-06). Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages. Quartz rebuilt (761 files emitted, 127 processed) and deployed topublic/;hermes-wiki-serve.servicerestarted. Script exit 0. ⚠ Threshold correction: the previously committedfreshness-20261106.txtcarried a 90-day threshold (the nightly audit's setting); the stocktake script's threshold is 21 days, and this run's report is the authoritative one for today. -
Freshness: 0 stale pages. The three no-date pages are unchanged (
SCHEMA.md,experiments/active/example.md,log.md). No page is over 21 days. -
Live-vs-page drift diff (the age scan alone would have missed all of this).
- infrastructure/tailscale - 1 state transition + 8 offline-age drifts + 2 dead Serve routes.
anthonys-mac-minicame back online (peer recordOnline: true;tailscale ping 100.76.60.37->pong ... via DERP(syd) in 103ms). Offline ages refreshed +1d each: cloudreve 103->104d, elementsynapse 106->107d, freshrss-new 79->80d, octoagent 3->4d, octop 1->2d, omarchybook 14->15d, omarchymac 3->4d, openmausbot 14->15d. Every peer IP still matches the live JSON exactly; no live peer missing; no orphan row.m93pre-pinged -> pong (3ms);octopre-pinged -> timeout, matching itsLastSeen. Container states cross-checked against livepct liston both hypervisors (pve 17 guests, m93p 19 guests + 2 VMs). ⚠/agent-estate-> 8769 and/synergy-prep-> 9123 are DEAD (no listener) while still present intailscale serve status- both routes now marked DEAD on the page, matching infrastructure/network-map which had already recorded them as dead on 2026-11-05. Every other Serve target has a live listener. - infrastructure/docker-services - the 2026-11-05 "the seeder does not exist on any host" finding was
FALSE; corrected on the page with strikethrough + dated correction. The
worldmonitor-seeder-*containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer,OnUnitInactiveSec=4h->worldmonitor-seeder-scheduled;worldmonitor-seed-fast.timer,OnUnitInactiveSec=15min->worldmonitor-seeder-fast), so they are visible only while a run is in flight. Live at 2026-11-06 03:1x both were present; live at 2026-11-05 05:1x neither was - hence the false negative. The "no worldmonitor-seeder cron job" claim was also wrong: the driver is a systemd timer pair, not cron. The earlier "rename" narrative was wrong too: one script, two timer-driven container names. CT200's 61-container running set still matches live set-wise (no drift). CT333's container renamed again:flamboyant_mahavira->nostalgic_keldysh(created 2026-11-05T22:15:57Z). CT201 20/26, CT460 1/1, CT1001 1/9, CT333 8/9 - all unchanged and correct. - systems/scheduled-tasks - no drift. Live
jobs.jsonreads 29 jobs / 26 enabled / 3 disabled, 1 non-cleanlast_status(066bb5191280Uber Eats Monthly Spend Report,delivery_failed, streak 0) - exactly what the page already records. - infrastructure/hosts - cron count 29/26/3 still correct (last re-verified 2026-10-31).
- infrastructure/tailscale - 1 state transition + 8 offline-age drifts + 2 dead Serve routes.
-
Updated current-state and index - the false seeder claim and the "every state matching" Tailscale claim are now marked superseded with dated corrections rather than silently rewritten (SCHEMA rule 9).
-
Not done / unchanged: no page removed or archived; no decision reversed; the open
deepseek-paid-smart-on-omniroutedispatch fault and the066bb5191280delivery failure carried, not re-diagnosed; the open secrets-at-rest incident not re-litigated. No secrets, tokens or keys written or quoted. -
Follow-up (04:0x AWST): the one page sitting exactly on the 21-day threshold was re-verified rather than left to expire.
incidents/2026-10-14-omniroute-node-connection-refused-storm.mdwas at 21d and would have tripped the threshold tomorrow. Its residual claim ("omniroute-1answers again") was re-probed live and still holds: TCP connect to100.78.63.20:20128succeeds (noErrno 111), the Tailscale peer readsOnline: true, and a keyedPOST /v1/chat/completionsreturns HTTP 200. Page re-verified (updated/verified_on-> 2026-11-06,review_after-> 2026-12-06) with the probe recorded. ⚠ New trap recorded on the page and in systems/scheduled-tasks: onomniroute-1the raw model iddeepseek/deepseek-v4-flashreturns HTTP 401No active credentials for provider: deepseek, while the fleet'sdeepseek-paid-smartalias returns HTTP 200 and resolves to that same model. The 401 is an alias-routing artefact, not an outage — probe the alias the fleet pins, never the raw model id. (A first probe with the wrongkey_envalso produced a 401; the correct key foromni-101isHERMES_CUSTOM_100_78_63_20_20128_API_KEY.)
[2026-11-07] audit - window 2026-11-06
Headline: a red row with no alert. 2026-11-06 settled 172 execution rows across 16 distinct jobs - 170
completed / 1 failed / 1 running (this audit). The single failure is a new, non-restart signature:
the Job Radar's 11:30 slot (5fc292ff0f3b, row 95347fa5…, 11:30:32 -> 13:26:44) was discarded as
stale on Fire claim ownership lost; stale result was discarded. because a direct fire of the same job
(08:27:09 -> 13:26:22, 4 h 59 m 13 s) was still holding the claim and delivered first. Every previous
occurrence of this string came from a restart/reload orphaning a live run; this one had no restart - the
claim was lost to another fire of the same job. The failed row raised no cron_incidents row (newest
ledger activity remains the 2026-11-05T12:36:17 closures), so nothing alerted. New page ->
incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.
- Ledger/day: deliveries 10 rows / 10
delivered/ 0for_failure; fleet 29 jobs (26 enabled / 3 disabled -Important-mail monitor,Rhino AgentMail Inbox Watcher,AgentMail Rhino Inbox Poll), the fifth consecutive audit at the same count, nothing added/retired/disabled/repinned, fleetupdated_at2026-11-07T00:39:46+08:00. Non-cleanlast_statusunchanged at 1 (066bb5191280,delivery_failed, streak 0). Unclosed incident rows 0 -> 0; the one open page remains incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts (untouched in-window - no rotation, deletion or scrub). - Job Radar recovered the day in full: the 14:30 slot ran a normal 12 m 27 s cycle and delivered at
14:42:59 -
SAME_DAY_STRONG_COUNT 4,VERIFIED_COVER_LETTER_COUNT 4,MISSING_COVER_LETTER_COUNT 0, cache total 494, 10 today-dated rows, 20 non-empty artifacts, Run status: COMPLETE. - The stocktake's overrun streak fully unwound:
1a8e90ea2271ran 03:00:32 -> 04:11:35 (1 h 11 m 03 s) against 7 h 40 m -> 5 h 41 m -> 3 h 02 m on 11-03/11-04/11-05. Stage 1 (dca8482e4f76) ran clean for a second consecutive day (07:00:44 -> 08:29:21, 1 h 28 m 37 s,suppressed), so the 11-04 Open entry about its two lost candidate sets is cleared in substance - the 11-03 ledger row closedresolvedat 2026-11-05T12:36:17 and both runs since have been clean. - The last audit's open question answered itself: the Mnemosyne Health Monitor's 11-05 12:00:45 slot (still in flight 12 h+ with no delivery at the last audit) completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered; its 11-06 12:00 slot then took 7 h 47 m 56 s and delivered too. Ballooned on consecutive days, never lost, no incident row - watching, not open.
- Sessions: 15 cron sessions started in-window (495 messages / 273 tool calls), 16 sessions with
activity (incl. one
desktopsession, 146 messages). Busiest:wiki-stocktake-maintenance157, Job Radar Discovery 92,Enhanced Weekend Adventure Planner82, Job Radar Write and Letters (14:30) 28, Mnemosyne Health Monitor 27. - Wiki maintenance:
wiki-maintain.shrun after this audit's writes (freshness reportraw/stocktake/freshness-20261107.txt); skills-index regenerated; Quartz rebuilt. Pages touched: systems/scheduled-tasks (+frontmatter, +Count bullet, +Re-verified 2026-11-07section), current-state (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident bullet), runbooks/job-radar (+concurrency subsection), this log, and the new incident page. Vault mirror/home/hermes/wikiupdated (both conventions).
2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
- Script:
wiki-maintain.sh(repo, canonical) run clean — skills-index regenerated (302 skills / 68 categories), freshness report written toraw/stocktake/freshness-20261107.txt, Quartz rebuilt (769 files) and deployed topublic/,hermes-wiki-serve.servicerestarted. - Freshness: 0 stale, 125 fresh, 3 no-date of 128 pages — no page is past the 21-day threshold.
(
log.mdhas noupdateddate and lands in the no-date bucket, which is correct for an append-only log.) Freshness is not accuracy, so a live-vs-page diff was run against every page carrying live claims. - Tailscale (infrastructure/tailscale.md): table rebuilt from
tailscale status --json. 1 state transition —octop(100.92.163.122) is back online (peer recordOnline: true;tailscale ping→pong from octop via DERP(syd) in 98ms); it is still no LXC or VM on either hypervisor, so it is an orphaned tailnet peer, not an estate change. 9 offline ages refreshed (+1d each). All 51 live IPv4 peers match the table's IPs exactly — no missing peers, no orphan rows, no IP mismatches (52 rows = 51 peers +hermes; the ~23 IPv6-onlyfunnel-ingress-nodepeers excluded by design). The CT460 Serve route table still matchestailscale serve statusexactly (12 paths on :443); both DEAD targets re-confirmed dead (no listener on8769/agent-estate, none on9123/synergy-prep), every other Serve target has a live listener, and/gitea(100.82.100.128:3000) answers HTTP 200. - Proxmox (infrastructure/proxmox.md): ⚠
m93pruns 20 LXCs, not 19 — CT107mem0was missing (running;mem0-mcp.serviceactive on:8890;pct exec 107 -- tailscale ip -4→100.73.94.53). Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads 20 LXCs — 18 running, 2 stopped (CT998, CT1264).pvestill matches exactly: 17 LXCs + 4 VMs. Both hosts run the identicalpve-manager 9.2.10build. - ⚠
m93puptime claim was wrong. The page read "up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"; liveuptime -sshows the host booted 2026-11-05 21:19:52 (up 1 day, 7:18, load 0.76 / 1.08 / 1.24 — normal). The 2026-10-30 reboot was real but belonged topve(boot 2026-10-30 16:24:00). - Storage refreshed on both hosts against live
pvesm status:pvessd076.43 → 78.29 % (~152 GiB free),SSD164.32 → 65.46 %,fred-qnap82.77 → 84.21 %,local79.66 → 81.62 %,local-lvm13.75 → 14.25 %;m93piMac shares +local60.25 → 76.21 %,nfs-proxmox-*82.78 → 84.21 %,nfs-ssd0-3976.43 → 78.29 %,omb-internal57.92 → 64.66 %. - Docker (infrastructure/docker-services.md): set-wise diff against live
docker pson CT200/201/333 — CT200 61 running / 68 total exact, CT201 20 / 26 exact, CT333 8 / 9 exact, but CT333's enumerated list still opened withflamboyant_mahavira, which is not running (live namenostalgic_keldysh, created 2026-11-05 22:15:57). Corrected. CT460ignisrunning. - Cross-page corrections: infrastructure/network-map
/giteawas annotated "Gitea (ct207 tailnet IP)" — it is CT221 onm93p(ct207 is100.82.40.23); also itsm93pPVE version read 9.2.0 (both hosts are 9.2.10). agents/hermes-chaos read "v0.22.3…5b147ab036d6— the same build as CT460", which contradicted itself: live isv0.22.5…bf0203a794a0, now genuinely identical to CT460's (hermes update --checkreports v0.23.0 available). agents/hermes-production's "2 cron jobs still pincustom:omni" is no longer true — livejobs.jsonhas 0 on the bare key and 21 oncustom:omni-101. - Fleet re-verified unchanged: CT460 29 jobs (26 enabled / 3 disabled); CT450 25 jobs
(13 enabled / 12 disabled) with 9 shared names — confirmed by a live set-diff of both
jobs.json. A keyeddeepseek-paid-smartchat probe againstomniroute-1(100.78.63.20:20128) returns HTTP 200;config.yamlstill defaults tocustom:omni-101. - Pages touched: infrastructure/tailscale, infrastructure/proxmox, infrastructure/hosts, infrastructure/network-map, infrastructure/docker-services, agents/hermes-production, agents/hermes-chaos, current-state, index, this log. No new incident page — no incident was found; every drift above was a documentation drift corrected against a live probe.
2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
- Anthony's ask: "fix this" on the failing backup alert — with a pointer to check Notion for the API key.
- incidents/2026-08-08-gitea-config-backup-push-dead is now
status: resolved. The nightlye27da6c34bbbconfig backup had been committing locally and failing to push every night since 2026-08-08 — 92 commits stranded — while its cron row readlast_status: ok. - Four independent root causes, all confirmed against live systems:
- No Gitea credential.
originwas HTTPS with no userinfo; the globalcredential.helper=storefile~/.git-credentialsheld only agithub.comentry. The token in the Notion vault was valid and byte-identical to the one the wiki repo carries in its remote URL — it simply was never wired into the backup repo. Fixed with a host-scopedgit credential approveentry (not embedded in the URL);git ls-remote originnow authenticates withGIT_TERMINAL_PROMPT=0. backup.shhad noexitstatement at all — it always returned 0, so the scheduler'ssuccess = (returncode == 0)recordedokon every failed night. The scheduler was correct; the script was the liar. This is why the outage ran 92 days without a job-level red.- A 1.76 GB
checkpoints/backlog in the unpushed range (13 blobs > 100 MB) — junk that.gitignorealready excluded for new commits but that history still carried. - Both secondary legs broken. The
githubremote did not exist, and the fallback pushed a branch's full ancestry, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file limit could never accept it. The AgentMail alert called apython3 send_email.pythat no longer exists, via achaos-agent@agentmail.toinbox that no longer exists.
- No Gitea credential.
- Backlog drained:
git filter-repo --partial --refs origin/main..main --path checkpoints --invert-paths --prune-empty never --forcestrippedcheckpoints/from the unpushed range only (origin/mainuntouched, push stayed a fast-forward), thencheckpoints/was untracked at the tip. Push set ~2 GB → 264.8 MB; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already onorigin/mainand never resent. Pre-rewrite history preserved under tagbackup/pre-rewrite-20261107-032658. Token leakage scan over the HEAD tree, the working tree and every unpushed blob: clean. - Script rewritten (
~/.hermes/config-backup/backup.sh, via OpenCode + a correction pass): exits 0 only on verified success; stamps.last_backup_okonly after remote verification; decides the push by remote-vs-local rather than by working-tree changes (the exact mechanism that created the backlog); guards everygit ls-remotewith|| trueso an unreachable remote reachesnotify_failureinstead of dying at exit 128 before the alert; pushes GitHub as a parentless orphan snapshot (git commit-treefromHEAD^{tree}— no history, no >100 MB blob, HEAD/branch/ index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery honestly (curl --fail+"ok":true; the realagentmailCLI, key read from.envat runtime and passed via the environment, never argv). - Verified live 2026-11-07: real run exit 0 in 47 s;
git rev-parse HEAD==git ls-remote origin refs/heads/main==a50a5170c1c0e2af5235fa63a95a732e2d780703;rev-list --count origin/main..main= 0; Gitea APImain=a50a5170c1c0, updated2026-11-07T05:10:46+08:00;.last_backup_okrefreshed (was stale from 2026-11-06 11:00). GitHub mirror now carries a 0-parent orphan snapshotfaed9fb5c677(30,086 files, largest blob 23.5 MB) withmasteruntouched atc99c94e6b13b— no data lost, since GitHub's tree was a strict subset of local HEAD (onlyBACKUP-INDEX.mdadded). AgentMail and Telegram alert paths each exercised directly (AgentMail self-test to the agent's own inbox; TelegramgetMe, no message sent to Anthony). Sandbox E2E: unreachable remote → exit 1 + alert + health not stamped; local-ahead-with-clean-tree → pushed + verified; already-up-to-date → no push, exit 0. - Pages touched: incidents/2026-08-08-gitea-config-backup-push-dead (status → resolved, full Resolution section added), infrastructure/backups (⚠ → ✅, re-verified 2026-11-07), systems/scheduled-tasks (job row → RESOLVED), this log.
- Note — unrelated pre-existing state: the 03:00
wiki-stocktake-maintenanceslot failed withRuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are included in this commit so the tree is not left dirty.
[2026-11-08] audit - window 2026-11-07
- Day ledger: 2026-11-07 closed 163 executions — 159
completed/ 3failed/ 1running(still open at probe time). Fleet unchanged: 29 jobs / 26 enabled. Deliveries 7 / 7delivered, 0for_failure. - New incident page: incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain. Two in-flight
cron executions lost their durable fire claims when a gateway restart drained at 05:00:43/47:
ea79a2107e36(Mnemosyne Health Monitor, 00:00:42.918 → 05:02:22.870) and1a8e90ea2271(wiki-stocktake-maintenance, 03:00:31.515 → 05:02:22.174). A restart requested at ~04:29:52 sat in after-turn mode for the full 1800 s (#77184), timed out, and the drain proceeded; systemd restarted the unit at 05:01:00 (restart counter is at 3). The thirdfailedrow of the night (96eb95ce12f2, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event ~19 s later. Request origin: inside theconduit_pushself-update window (04:27:55 → 04:32:32, whose own ledger recordsrestarting_gateway= failed) — stated on the page as a working hypothesis, not proven; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13). - Incident ledger: unclosed rows 0 → 2, and neither alerted. Both are the stocktake job's:
1a8e90_628d914f1f97(first seen 04:29:54.815) and1a8e90_2f370ef764f5(05:02:21.573), bothdetectedwithalerted_at: null. The health monitor's two rows closed 15:43:17 / resolved 17:08:52. - Hermes 0.22.5 → 0.23.0 on CT460 (
hermes update, 05:05:13 → 05:43:10,bf0203a794a0→2bd378ed1b8d); thesnapshotstage was skipped again (updates.pre_update_backupmodeoff) → still no rollback point. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (livepid 3109529). CT450's install stamp now also reads 0.23.0. - Plugin/credential state changes:
camofoxv1.1.0 is now an enabled plugin (browser backend is no longer core);conduit_pushself-updated 0.122.2 → 0.134.4 (tree HEADef3d2f2);hermes-hq-pushreinstalled 0.9.27 → 0.9.42 (upstreamd26850e7, 60/60 files, tree mtime 2026-11-07 06:42) with its routes only loaded by the 11:11 restart; the OpenCode Go key is expired — HTTP 401api_key_expiredon…/zen/go/v1/models, probed twice in-window, with no 2026-11-07 execution failing on an auth error. - Secrets incident re-checked, still OPEN: a regex sweep of every
cron/jobs.jsonprompt for secret-shaped strings returned 0 hits, butconfig.yaml'sredact_secrets:value could not be read as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed. - ⚠ Skill inventory moved: 302 → 299 skills, 68 → 67 categories. Four skills moved into
~/.hermes/skills/.archive/in one pass at 2026-11-07 07:43 (cron-model-repair,fleet-wide-failure-triage,hermes-instance-update,mail-lookup); parked archive 46 → 50, and theCron Model Repaircategory disappeared with its only member. A grep of every activeSKILL.mdfinds no absorbing skill, so the "absorbed into X" pointer is missing.microsoft-officeappeared in-window (Productivity 25 → 26). Disk shows 300SKILL.mdvs 299 listed (generator de-dups one nested copy). - Freshness: 0 stale / 126 fresh / 3 no-date of 129 pages —
raw/stocktake/freshness-20261108.txt. - Pages touched: incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain (new),
systems/hermes-agent, systems/scheduled-tasks, systems/browser-backend, systems/conduit,
systems/hermes-hq-push, systems/opencode-go, systems/skills-index (regenerated),
incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts, agents/hermes-production,
agents/hermes-chaos, current-state, index.md, this log, and the vault mirror
(
/home/hermes/wiki). - Wiki maintenance:
wiki-maintain.sh(canonical repo script, via the~/.hermes/scripts/wrapper) run after this audit's writes — skills-index regenerated (299 / 67), freshness report written, Quartz rebuilt (129 markdown → 779 files deployed topublic/) andhermes-wiki-serve.servicerestarted at 2026-11-08 00:38:29 AWST. - Wiki maintenance:
wiki-maintain.sh(canonical repo script) run after this audit's writes —systems/skills-index.mdregenerated (299 skills / 67 categories; disk shows 300SKILL.md, one nested duplicate de-duped by the generator), freshness reportraw/stocktake/freshness-20261109.txt(0 stale / 127 fresh / 3 no-date of 130 pages), Quartz rebuilt (130 markdown → 793 files deployed topublic/) andhermes-wiki-serve.service(a user unit —systemctl --user) restarted 2026-11-09 00:18:30 AWST. Published pages verified HTTP 200 athttp://127.0.0.1:9120/<path>and on the external routehttps://hermes.kangaroo-eel.ts.net/wiki/<path>. - No wiki page removed or archived. No job added, retired, disabled or repinned; no system obsoleted; no decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
- No secrets, keys, tokens or passwords written — the opencode-go finding records the HTTP status and the upstream error code only.
[2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day
- New page:
incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md— RESOLVED same day. - The finding:
conduit_pushis the onlypre_tool_callregistrant on the host (plugins/conduit_push/__init__.py:119), and its callback overran Hermes's built-in 30 s hook budget 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03 —~/.hermes/logs/errors.logholds 12 lines on 11-07 and 46 on 11-08, plus 89 "skipped after previous timeout or while still running" suppression lines. Every tool call degraded from slow to suppressed. - The fix (2026-11-08 13:35 AWST):
plugins.hook_callback_timeout30 → 120 in~/.hermes/config.yaml. Verified live, and verified effective — the last timeout line is still 08:21:03, re-checked at audit time. - A plausible diagnosis measured and rejected: the plugin's caching was not the fault. Timed
in-process: 3.1 s cold, 0.00 s warm (
session_titles.py:21 CACHE_SECONDS = 30.0). The "re-opens a 1.87 GBstate.dbper call, ~9 s each" theory was wrong; the trigger is load on the 2-core CT460 (load average 5.89at 13:27:50;state.dbcheckpointing a 40 MB WAL on the same spindle). Next lever isstate.dbcheckpoint/retention discipline, not more caching. - Cost of the fault: Mnemosyne Health Monitor ran 00:01:20 → 08:29:05 (8 h 27 m) and
12:00:51 → 18:00:42 (5 h 59 m 51 s) — its two slowest slots on record, both delivered; FreshRSS Midday
Briefing
ecc1fc13655atook 2 h 09 m 53 s; and the stocktake failed a third day. - Cron ledger: 11-08 settled 169 rows — 168
completed/ 1failed/ 1running(this audit, 23:59:33). The single failure is1a8e90ea2271(wiki-stocktake-maintenance), 03:01:11 → 04:36:08, on a new terminal signature:RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.(the previous two red days read "lost its durable fire claim ownership").failure_streak: 3. Working hypothesis, not proven: a suppressedpre_tool_callholding a tool result open is exactly what the error describes, but the string does not name the hook. Deliveries 8 rows / 8delivered/ 0for_failure. Fleet 29 jobs (26 enabled / 3 disabled) — a sixth consecutive audit at the same count, nothing added, retired, disabled or repinned (fleetupdated_atmoved to2026-11-09T00:06:50+08:00, read as a run-state stamp only). - Incidents: unclosed rows 2 → 3 — a third silent
detectedrow for the stocktake (1a8e90_a9cad99cb150,alerted_at: null), all three still unreconciled. The alerting path is not at fault: the Newsletter Digest row85edf6_0877ca1e0eacalerted 07:23:07 and closedresolvedat 08:06:54 (its admission-readdatabase is locked, cleared when the run completed). - ✅ Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works. The 18:00 slot
(agent mode) ran 12:00:51 → 18:00:42 = 5 h 59 m 51 s; the converted job's first slot ran
2026-11-09 00:00:48 → 00:02:02 = 73 s, exit 0, silent, writing a 159 B output file against
105,982 B for the last agent-mode run. Coverage live at working 1030/1030 = 100.0 %, episodic
1192/1192 = 100.0 %, so the
< 95 %alert is correctly quiet. The script compares withawkbecausebcis not installed on this host. - ✅ Verified resolution 2 —
state.dbre-measured and one corruption remnant removed. Now 1,890,656,256 B (1.89 GB), 2,901 sessions / 197,682 messages, live 40,157,672 B WAL.state.db.corrupted.20260811_151052(624.1 MB + 32 KB-shm+ 0 B-wal) deleted in-window;state.db.recovered.recovery.json(14,566 B) remains andmnemosyne/data/mnemosyne.db.corrupted.20260809_120404(23.6 MB) was not cleaned. - Pages updated:
systems/conduit.md(the pre-tool-call section, total),systems/hermes-agent.md(hook budget +state.dbsize, total),systems/scheduled-tasks.md(new Count bullet, total),systems/mnemosyne.md(health-monitor conversion verified, total),current-state.md(total),incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md(state.db1.7 → 1.89 GB in three places, reconfident),incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md(open items),index.md(new entry + incident list +state.dbfigure, confidence dropped), and this log. - No wiki page removed or archived. No system obsoleted; no decision reversed.
- No secrets, keys, tokens or passwords written — no credential value appears anywhere in this audit.