68 KiB
Wiki Log
Chronological record of wiki actions. Append-only.
[2026-07-22] init | Fresh Hermes Systems Wiki
- Created technical wiki at
/home/hermes/wiki - Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
- Wrote SCHEMA.md with 11 operating rules
- Wrote index.md as curated Start here dashboard
- Reseeded system/infrastructure/runbook stubs for existing homelab
- Git initialized, local-only
- No secrets, passwords, tokens, or private keys written to vault
[2026-07-22] audit | External audit pass (Claude, at Anthony's request)
- Merged duplicate pages:
systems/omniRoute.md(empty stub) -> keptsystems/omniroute.md(populated); repointed 5 links - Merged duplicate pages:
runbooks/safely-update-hermes.md-> kept asrunbooks/update-hermes-safely.md(fuller content); repointed 2 links - Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
- Fixed a broken
review_after: 0001-01-01in decisions/keep-openrouter-as-fallback.md - New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
[2026-07-25] operations | Tool Search and gateway resilience
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
- Documented how new skill categories are derived from the first folder below
~/.hermes/skills/ - Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
- Recorded the I/O-controller delegation limitation so an ineffective
IOWeightis not mistaken for active protection - Added health checks, exact effective settings, backup location, and rollback procedure
- No secrets, tokens, or private credentials were written to the wiki
[2026-07-28] dashboard | Jobs Dashboard feature additions
- Fixed bulk status update (payload shape mismatch:
{ids, status}→{ids, updates: {status}}) - Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion
Notes / JD Summary - Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
- Updated Synergy and Dept of Communities to Phone Screen status
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
[2026-07-27] incident | Job Radar search and dashboard recovery
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require
MISSING_COVER_LETTER_COUNT=0 - Recorded the production cron model (
gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5) - Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both
:443Serve and:8443/webhookFunnel - Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
- Verified no secrets, API keys or private credentials were written to the wiki
[2026-07-29] update | Steel Browser deployment — browser backend switch
- New page:
systems/browser-backend.md— rewritten to reflect Steel Browser as primary, Camoufox as fallback - New page:
comparisons/steel-vs-camoufox.md— comparison of the two backends - Updated:
infrastructure/hosts.md— added CT201 and CT460 to verified hosts table - Updated:
infrastructure/docker-services.md— added Steel Browser compose file - Updated:
infrastructure/tailscale.md— added CT201 Steel serve routes - Updated:
index.md— browser-backend link description - Updated:
current-state.md— added 2026-07-29 changes section - Config change:
browser.cdp_urlset tohttp://100.96.244.39:9223in ops profile
[2026-07-30] update | Open WebUI updated on CT245
- Updated image to
ghcr.io/open-webui/open-webui:main(new digest e97bf9531916, Jul 27) - Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
- Restarted container after OOM hang
- Updated
infrastructure/proxmox.md
[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
- Ran
mnemosyne reindex --yesto rebuild all embeddings — working memory now at 100% - Updated health check script (
mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py) - Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
- DB grew from 8.7M to 21M with the new embeddings
[2026-08-01] incident | Mnemosyne capture pipeline repaired
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no
working_memoryrow existed. - Corrected stringified structured YAML (
sync_roles,skip_contexts,reflect,sleep_threshold,ignore_patterns) and enabled profile isolation;hermes config checkpassed. - Verified the repair with a second CLI marker that persisted as an exact normalized
[USER]database row. - Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route
:9119; added authoritative identity/preference/workflow slots. - Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
- Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.
[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
- Rewrote comparisons/steel-vs-camoufox — full Steel experiment story, timeline, lessons learned
- Rewrote systems/browser-backend — Camoufox primary, Steel disabled
- Updated current-state — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
- Updated concepts/browser-automation — added Steel experiment lesson
- Created decisions/rollback-steel-to-camofox — formal decision record
- Updated systems/scheduled-tasks — job count 25, Uber Eats upgrade, GH Trending Daily removed
- Updated systems/mnemosyne — vector embedding coverage gap documented
- Updated systems/model-providers — DeepSeek V4 Flash 0731 details
- Created reports/uber-eats/index — landing page for monthly spend reports
- Updated decisions/index — linked new decision page
- Updated index.md — browser description, reports section, last-updated date
- Created nightly wiki audit cron job (23:59 AWST)
[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
- Ops profile (
~/.hermes/profiles/ops/config.yaml): enabledmnemosyneplugin withprofile_isolation: true,allow_tool_override: false,shared_surface_read: false - Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
- Verified end-to-end: memory writes land correctly in both profiles independently
- Updated systems/mnemosyne — profile isolation section now documents ops config
- Updated current-state — added ops profile Mnemosyne entry
- Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
- Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision
[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on
deepseek-flash-freeandsmart-route/good-freeacross both instances - Reworked
deepseek-flash-freeandgood-freeon CT600: strategypriority→fill-first, dropped proven culprits (238d5978/deepseek-v4-flash+-0731,nvidiadeepseek), healthy free models promoted, deepseek demoted to last resort - Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
- Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
- Updated systems/omniroute — added Recent Changes (2026-08-04) section
- Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated
updateddate
[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
- OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern.
is_active=0does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill. - Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
- Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
- Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
- Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
- Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
- Updated index.md — last-updated date 2026-08-04 → 2026-08-07
[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
- Portainer password reset documented: CT245 admin user confirmed as
anthony(notadmin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state. - SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint,
//root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.
[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
- Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from
freeapi/minimax-m2.7→smart-route(IDs:c8d52904e1d6,1b23c042e17d,d40085da631c). Mosquito Tracker fromfreeapi/auto→smart-route(ID:1b31187397a5). 9 stale-named jobs (e.g.vertex/gemini-2.5-flash,deepseek-v4-flash-free) →smart-route. Daily Marketing Job Radar →gpt-5.6-terra. - Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted
hermes-ui.service. Production build serves/ui/path correctly with relative asset paths.hermes-desktop.martinwa.orgidentified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460. - Phase One estate review completed: Manual cron run (ID
a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled withlast_status=ok, CT333 Nanobot has livejobs.jsonwith 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes. - Updated current-state — added 2026-08-08 changes section
- Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues
[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
- Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
- Root cause identified: Disk at 83% (11GB free on 65GB),
state.dbat 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues. - Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
- Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
- OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
- No new pages created — all changes applied to existing pages
[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
- OmniRoute AutoRoute variants documented:
auto/best-free(Auto Cheap) andauto/coding-freeare virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed:auto/best-freerouted to Antigravity. Updated systems/omniroute. - OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains
memory.provider: mnemosyne. Added to comparisons/memory-backends. - FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default
freellmapi-auto. Added to comparisons/model-routers. - No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.
[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
- OmniRoute migrated from CT600 to CT601: Provider renamed
custom:omniroute→custom:omni; base URLhttp://omniroute:20128/v1→http://omni:20128/v1. Hostomni.kangaroo-eel.ts.net(100.93.204.4, active) replacesomniroute.kangaroo-eel.ts.net(100.88.81.19, offline). CT600 stopped, CT601 running. DB path/var/lib/omniroute/storage.sqlite→/opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22)custom:omnirouterefs repointed tocustom:omni. Pre-migration backup at~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list,grep, Tailscale DNS). - Updated systems/omniroute — Current Configuration rewritten for
omni/CT601, added Recent Changes (2026-08-13), bumpedupdated/verified_onto 2026-08-13. - Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped
updated. - Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
- Updated infrastructure/hosts — OmniRoute proxy entry →
omni; CT600 reference marked verified/retired. - Updated index — last-updated date 2026-08-10 → 2026-08-13.
- No new pages created; all changes applied to existing pages. No secrets written.
[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
- Skills index drift incident:
systems/skills-index.mdclaimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creatingregenerate-skills-index.shscript, and adding cron job1a8e90ea2271(daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift. - Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped
updated/verified_onto 2026-08-15. - Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
- Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
- Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
- Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
- No secrets written.
[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
- New page:
systems/nextcloud.md— documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages. - Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
- Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
- Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
- No secrets written.
[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
- OmniRoute upstream capacity pressure documented:
ServiceUnavailableError: Structurally heavy chat request capacity is busyreported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded. - Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
- Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config
http://omni:20128/v1is correct —/v1is required. - Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped
updated/verified_onto 2026-08-16. - Updated current-state — added Recent Changes (2026-08-16) section, bumped
updatedto 2026-08-16. - Updated index.md — bumped last-updated date to 2026-08-16.
- No new pages created. No secrets written.
[2026-08-18] incident | Mnemosyne capture pipeline broken (active)
- New page:
incidents/2026-08-18-mnemosyne-capture-pipeline— working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions inrun_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification. - Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped
updated/verified_onto 2026-08-18, added cross-reference to Related. - Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped
updatedto 2026-08-18. - Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
- Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from
activetoresolved(that earlier incident was fixed). - No secrets written.
[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
- Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
wiki-stocktake-maintenance(1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed fromcx/gpt-5.6-luna→go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.- CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
- Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
- Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped
updated/verified_onto 2026-08-23. - Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped
updated/verified_onto 2026-08-23. - Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
- Updated index.md — bumped last-updated date to 2026-08-23.
- No secrets written.
[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
- New page:
systems/ignis.md— documented Ignis browser-native Obsidian (Dockerignis-subpath:0.8.9, port 8080, Tailscale Serve athermes.kangaroo-eel.ts.net/ignis). Volume mount:/home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active. - Updated systems/model-providers — added
command-codeprovider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumpedupdated/verified_onto 2026-08-19. - Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider,
deepseek-flash-mixcombo, and Tavili search proxy status. Bumpedupdated/verified_onto 2026-08-19. - Updated infrastructure/docker-services — added Ignis compose file and service definition.
- Updated infrastructure/hosts — added Ignis to CT460 row.
- Updated systems/obsidian — added Ignis as browser interface, corrected vault path to
agent-estate-wiki/, added Quartz static site reference. - Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
- Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
- No secrets written.
[2026-08-26] update | Nightly wiki audit — Cron job reconciliation
- Updated systems/scheduled-tasks — reconciled live
hermes cron list(27 jobs) against wiki table (29 entries). Removed:Workday start reminder(no longer in live list),CT 221 Miner Watchdog(jobd5516d0a5147removed; CT221 status unknown),Review Agent Estate Phase One(one-time job completed 2026-08-08). Added:weekly-save-restricted-bot-restart(0 3 * * 0, local, no-agent). Updated count from 29 to 27. - Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
- Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
- Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
- No secrets written.
[2026-09-02] incident | SFTPGo outage on CT299
- New page:
incidents/2026-09-02-sftpgo-outage.md— documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%),tailscaleddied; PVEpct startfailed due to 5 stalelxc-attachprocesses pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148,pct stop 299 --skiplock+pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP bannerSSH-2.0-SFTPGo_2.7.0. - Updated current-state — added Recent Changes (2026-09-02) section, bumped
updated/verified_onto 2026-09-04. - Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
- No secrets written.
[2026-08-28] update | Department of Communities role and social-media audit
- New page:
reports/communities-wa-social-media-audit-2026— durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role. - Updated systems/current-profile — recorded current employment context and linked the report.
- Updated index.md — added the report to the Reports section and refreshed the last-updated date.
- No secrets written.
[2026-09-20] update | Nightly wiki audit — No changes in past 24h
- No new pages created — no new systems, decisions, or incidents in the past 24 hours.
- No existing pages updated — all wiki content remains current.
- Freshness audit run:
freshness-20260920.txtgenerated at 03:01 AWST (scheduled cron). - Skills index auto-regenerated:
systems/skills-index.mdupdated at 03:01 AWST (scheduled cron1a8e90ea2271). - Scheduled tasks reconciled:
systems/scheduled-tasks.mdupdated at 03:02 AWST — 26 active jobs verified from livehermes cron list. - No secrets written.
[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
- New page
incidents/2026-09-28-hermes-managed-runtime-deps— the 2026-09-28 → 30 cron-fleet outage: jobs failed withRestart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused byModuleNotFoundError: No module named 'dotenv'/'mnemosyne'in the uv-managed runtime venvs under~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2 (all 10 venvs,pip install --target), run every 15 min by job96eb95ce12f2. - Updated
systems/scheduled-tasks.md— table rebuilt from live~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected tod40085da631c,f77b7e46bd09,0a7f933d06e2. Known Issues refreshed: job-radar 403MODEL_NOT_IN_PLAN(streak 5), Telegram thread-delivery failures,weekly-save-restricted-bot-restartscript restored, residual pre-fix dispatch errors oncbfe434a3b5e/1a8e90ea2271. - Updated
systems/hermes-agent.md— version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/modelcustom→http://100.78.63.20:20128/smart-route; skills 330 → 341; cron count 29 → 30. - Updated
current-state.md—_config_version: 46and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section. - Updated
runbooks/job-radar.md— corrected primary model row (custom:omni-101 / gpt-5.6-terraviacommand-code) and added the 403MODEL_NOT_IN_PLANfailure mode with recovery steps. - Updated
systems/messaging-integrations.md— Telegramallowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup). - Updated
index.md— last-updated to 2026-09-30, cron count to 30, new incident listed. - No secrets written.
[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
- No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
- Updated
systems/scheduled-tasks.md— table and counts re-verified against live~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 —dca8482e4f76(403MODEL_NOT_IN_PLAN, ×5),1a8e90ea2271(HTTP 410, model retired upstream, ×7),fad5a86979d4(HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes:1b23c042e17dandc8d52904e1d6delivery targets nowtelegram:-1004321904721:102;c0a9fa314542flagged for a revoked Telegram bot token; bare-telegramtargets noted on0e753e9720fd,fad5a86979d4,066bb5191280. Known Issues restructured into Resolved / Open / Historical. - Updated
systems/messaging-integrations.md— new Delivery target semantics section: a baredeliver: "telegram"resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails withMessage thread not foundand the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs. - Updated
systems/model-providers.md— provider stack rebuilt from liveconfig.yaml: 5 custom providers (omni-1015,384 models,Omniroute:201285,398,litellm1,728,command-code71,guacano46);custom:omnirecorded as removed. Default model correctedresilient-free→smart-route. New Retired / Unavailable Models table (retiredqwen3.5:397b, plan-gatedgpt-5.6-terra, catalog-missingdeepseek-v4-flash) with the lesson that a raw model pin never self-recovers. - Updated
systems/mnemosyne.md— new Known Issue: gateway background-process notifications were being written into the person-attributed tier as[USER] [Anthony Martin]memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy. - Updated
systems/hermes-agent.md— MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins. - Updated
index.mdandcurrent-state.md— last-updated bumped to 2026-10-01 with a Recent Changes block. - Vault sync: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live underwiki/, not at the tree root, andindex.mdsits at the vault root. - No secrets written.
[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split
- New page:
incidents/2026-10-01-cliproxy-connection-auth-expired.md— job06a0a1c8704f(AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run withHTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard(providercustom:Omniroute:20128, modelcliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d,79f9bb41ecf5). - Updated
systems/scheduled-tasks.md— counts re-verified against live~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01:06a0a1c8704f(HTTP 401),0e753e9720fd(HTTP 400 — same NVIDIA model pin as the already-failingfad5a86979d4, confirming the shared-pin diagnosis),066bb5191280(delivery failed; report itself generated). Streaks advanced:dca8482e4f76×5→×6,1a8e90ea2271×7→×8,fad5a86979d4×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees. - Updated
systems/messaging-integrations.md— corrected a wrong page fact. The page previously stated that a badthread_idalways loses the message. The 2026-10-01 records show two variants:Message thread not found(hard failure, message lost — the 2026-09-30 case) andconfigured thread_id <n> for telegram:<dm> was not found; delivered without thread_id(a thread-less fallback send does occur, yet the run is still recordeddelivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegramjobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today". - Updated
systems/current-profile.md— full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no~/.hermes/profiles/default/directory (the default profile is the top-level config;opsis the only named sub-profile), the model default issmart-route(notfreellmapi-auto), the default provider iscustom→http://100.78.63.20:20128/ OmniRoute (litellm is now only the last fallback entry), and the config is_config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings. - Updated
current-state.md— new Recent Changes (2026-10-02) block; frontmatter dates bumped. - Updated
index.md— last-updated to 2026-10-02; cron line now notes the failing count. - Structural finding — the wiki exists as two trees and they have drifted both ways.
/home/hermes/workspace/agent-estate-wikiis canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is whatwiki-maintain.shcommits./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy the iPad reads (pages underwiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions —systems/ignis.md,systems/nextcloud.mdand threeincidents/pages exist only in the vault, whileraw/stocktake/*,agents/,incidents/2026-09-02-sftpgo-outage.mdandreports/communities-wa-social-media-audit-2026.mdexist only in canonical — so canonicalindex.mdcurrently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees. - No secrets written; all credential-shaped values in config remain redacted.
[2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
- Updated
systems/scheduled-tasks.md— fleet re-verified against live~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), failing set still 6 but the composition moved.1b23c042e17d(Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the identicalHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboardthat the cliprox incident predicted — making it 2 of the 3cliprox/terrajobs confirmed broken, which retires any "one-job fluke" reading.79f9bb41ecf5is the last untested job on that pin (next due 2026-10-04 18:00). Streak recount againstexecutions.db:dca8482e4f76×6→×7, but1a8e90ea2271×8→×7 — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). Resolved:1b23c042e17d's staledelivery_failedflag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). Downgraded: the GitHub Trending Radar revoked-token claim is now "unverified at last run" —c0a9fa314542's 2026-09-30 19:22 run completed and delivered. New Known Issue recording that a delivered notice is not a successful job (both cliprox failures readdelivery_outcome: deliveredon afailedstatus). - New page
incidents/2026-08-26-work-outlook-basic-auth-disabled.md— theworkOutlook mailbox has been unreachable since 2026-08-26: himalaya is configured for password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workdies withlogin not supported. TCP/TLS tooutlook.office365.com:993is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap. - Updated
systems/messaging-integrations.md— Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable,workunreachable; frontmatterupdated/verified_onbumped. Added the tooling consequence:email/mail-lookupresolves the account before searching, so work-mail requests hit this wall and must be reported as account unreachable, not "no results found". - Updated
systems/hermes-agent.md— skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to six jobs across three model faults plus one credential fault, and a second entry added for the stock-take job being down; theEmail: connectedbullet corrected. - Updated
current-state.mdandindex.md— new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped. systems/skills-index.mdregenerated 341 → 344 — including a new skill,fleet-wide-failure-triage(devops). This was done by hand: job1a8e90ea2271is still blocked on HTTP 410 (retiredqwen3.5:397b) and cannot self-recover.wiki-maintain.shitself is healthy and was run manually, also writingraw/stocktake/freshness-20261003.txt— recorded so the next reader knows the stocktake is on manual refresh, not on schedule.- Vault mirror: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live underwiki/;index.mdsits at the vault root. - No secrets written; all credential-shaped values in config remain redacted.
[2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST.
New pages
incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md—mnemosyne_recallhad been scoring keyword-only:fastembedwas absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 without raising an error. Repaired 2026-10-03; the runtime guard now asserts fastembed +FASTEMBED_CACHE_PATH+ model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%).incidents/2026-10-03-ct450-hermes-update.md— CT450 (hermesagent) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (ssh root@100.96.100.82→pct exec 450 -- …) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms).
Updated
current-state.md— new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and themnemosyne doctor/sqlite_vecmisread gotcha.systems/mnemosyne.md— Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage jobb63ed5817212documented.systems/hermes-agent.md— version corrected tov0.21.5+6169.gab86a20(wasv0.21.5+3837.gbfda74c), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (noprofiles/default/;opsis the only named sub-profile), Known Issues bullet added.systems/memory-backend.md— Known Issues bullet for the FTS-only degradation.systems/scheduled-tasks.md— 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window),79f9bb41ecf5due 2026-10-04 18:00; bare-telegramdelivery warnings confirmed live.agents/hermes-production.md— version, Python, profile, provider routing, cron count and messaging refreshed; stalecustom:litellm/freellmapi-autoreferences replaced.agents/hermes-chaos.md— CT450 version bump recorded, no-SSH access path added, Mattermost failure noted.infrastructure/hosts.md— CT450 row updated to v0.21.5.runbooks/update-hermes-safely.md— new "Second instance — CT450" section (access path + pre-flight checks).runbooks/provider-health.md— stale v0.19.0 version reference corrected.index.md— last-updated bumped; both new incidents linked; cron-fleet line re-dated.
Note — mnemosyne doctor on this host prints vec0: present_but_unloadable and marks the vector tiers unavailable. That is a Python-3.11-legacy-venv artefact, not the gateway's state. Judge vector health with ~/.hermes/scripts/check_embedding_coverage.py.
[2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST.
- Updated
systems/scheduled-tasks.md— fleet re-verified against live~/.hermes/cron/jobs.json+hermes cron list: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 6 → 7. The one new failure event in the window is79f9bb41ecf5(Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identicalHTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. It was the last untested job on thecliprox/terrapin, so that pin is now 3 of 3 confirmed broken — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271) and the Known Issues Open list all updated; frontmatter dates bumped. - Updated
incidents/2026-10-01-cliproxy-connection-auth-expired.md— new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still Open — needs the OmniRoute connection re-authenticated before any repin. - Updated
current-state.md— new Recent Changes (2026-10-05) block: the cliprox 3/3 settlement,1a8e90ea2271's repeat 410 on 2026-10-04, confirmation that no interactive sessions occurred in the window (all 10 active sessions were cron), and a streak-counting caveat. - Updated
index.md— last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed". - Streak-number caveat recorded:
hermes cron list's "N failures in a row" is thefailure_streakcounter injobs.json, not a DB count.~/.hermes/cron/executions.dbnow retains only rows from 2026-09-30 onward (1,001 rows total), soexecutions.dbrecounts undercount older streaks — the earlier DB-based "×7" for1a8e90ea2271undercounts against the live ×11. Quote the live counter and note the retention floor. - Vault mirror: the same page edits were mirrored into
/home/hermes/wiki/agent-estate-wiki/(Obsidian/Ignis vault) so the iPad view matches the Gitea source —wiki/systems/scheduled-tasks.md,wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md,current-state.md,index.md,log.md. - No secrets written; all credential-shaped values remain referenced by connection ID only.
[2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0
Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST.
New
incidents/2026-10-05-dead-model-pin-repin.md— the seven-job dead-model-pin repin. Records the four dead pins (cliprox/terraHTTP 401 ×3,ollama-cloud/qwen3.5:397bHTTP 410 ×1,nvidia/deepseek-ai/deepseek-v4-flashHTTP 400 ×2,command-code/gpt-5.6-terraHTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (free-stack×6,smart-route×1 oncustom:omni-101) over another raw model name.
Updated
systems/scheduled-tasks.md— Count block rewritten: 0 failing (was 7), with all 30failure_streakcounters at 0; four jobs carry a stale pre-repinlast_status: errorthat clears on their next scheduled run. Seven job-table rows updated (06a0a1c8704f,1b23c042e17d,79f9bb41ecf5,1a8e90ea2271,fad5a86979d4,0e753e9720fd,dca8482e4f76), plusc0a9fa314542. New### Resolved 2026-10-05subsection in Known Issues; five resolved bullets removed from### Open, leaving bare-telegramtargets, the two-wiki-tree drift, the work-mailbox block, the stalecustom:omnikey and the delivered-notice rule as the standing open set.index.md— new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed.current-state.md— new Recent Changes (2026-10-06) block; frontmatter dates bumped.
Findings worth the ink
dca8482e4f76(Daily Marketing Job Radar) delivered to topic 1030 at 23:56 — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for79f9bb41ecf5(23:39) and06a0a1c8704f(23:45).- GitHub Trending Radar's revoked-token diagnosis is dead.
c0a9fa314542ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. NogetMe→ 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright. delivery_outcome: deliveredstill does not mean the job worked. Every pre-repin failure recordeddelivered, because the failure notice is itself the deliverable. Judge on the executionstatuscolumn.- A stale notice opened the review. Anthony was prompted by a
85edf6772bcanotice that had sat in topic346since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice. - Window tally (
executions.db, 2026-10-05 00:00 → 2026-10-06 00:15): 185completed, 3failed(all pre-repin, 03:00–07:00), 2unknown(aborted first attempts of79f9bb41ecf5anddca8482e4f76at 23:33/23:48, superseded by their successful re-runs) and 1running(this audit).
Vault mirror: the new incident page plus systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
[2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST.
Updated
systems/scheduled-tasks.md— Count block corrected: the stale pre-repinlast_status: errorset is now three (1b23c042e17d,fad5a86979d4,0e753e9720fd), not four —1a8e90ea2271cleared its own when it ran at 03:00. Added a Re-verified 2026-10-07 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176completed/ 0failed/ 1running; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended.index.md— last-updated → 2026-10-07; cron-fleet line re-dated (0 failing as of 2026-10-07, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration.current-state.md— new Recent Changes (2026-10-07) block; frontmatter dates bumped.
Findings worth the ink
1a8e90ea2271ran itself at 03:00 and proved the repin.wiki-stocktake-maintenancecompleted 2026-10-06 03:00:31 → 03:03:32 oncustom:omni-101/free-stack— its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retiredqwen3.5:397b). It regeneratedsystems/skills-index.md(344 → 346 skills), wroteraw/stocktake/freshness-20261006.txtand committed to Gitea as33f4e95. Its four HTTP-410 incidents (6df957a05ec6,d284168d3663,cda67de5b3e3,e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. The daily manualwiki-maintain.shrun can stop.- Vault mirror drift found and partly repaired.
systems/skills-index.mdin/home/hermes/wiki/agent-estate-wiki/was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item. - Window tally (
executions.db, 2026-10-06 00:00 → 2026-10-07 00:00): 176completed, 0failed, 1running(this audit) — 177 rows. 14 sessions, allcron; no interactive (non-cron) sessions, and no new incident opened.
Vault mirror: systems/scheduled-tasks.md, systems/skills-index.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by connection ID only.
[2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings
Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST.
Updated
systems/scheduled-tasks.md— new Re-verified 2026-10-08 AWST bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176completed/ 1failed/ 1running; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New Resolved 2026-10-07 Known Issues section, plus two new Open items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs.index.md— last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat.current-state.md— new Recent Changes (2026-10-08) block; frontmatter dates bumped.
Findings worth the ink
- The incident machinery had its first real workout since 2026-09-30 — and it worked.
62619aa6bcb3(DHL Tracking) failed its 08:00 run insideensure_browser()withTimeoutError: timed outon a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident62619a_f485ccfd4a0d(failure_type: timeout) was opened 08:00:52, alerted 08:00:58 and auto-resolved 11:01:13 when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed. - The tracked parcel has arrived. The 14:00 run reported waybill
8052838382asDelivered, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed. - Stocktake ran automatically for the second consecutive day (
1a8e90ea2271, 2026-10-07 03:00, commit8c63ef2; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once. - Two real defects surfaced from the stocktake's own log. (1) Step 3 of
wiki-maintain.sh— the Quartz rebuild — is skipped on every run: there is noquartz.config.tsand nonpx, sopublic/is never refreshed and the published site silently ages. (2)skills-indexunder-reports by one: 347SKILL.mdfiles exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicateproductivity/obsidian-git-sync/obsidian-git-sync/SKILL.md(the other duplicated basename,hermes-api-vault, survives because its two copies sit in different categories). Both are now Open items in systems/scheduled-tasks. - Unchanged, still carrying a stale
last_status: error:1b23c042e17d,fad5a86979d4,0e753e9720fd— due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-telegramtargets and thecustom:omnistale-key references are also unchanged.
Vault mirror: systems/scheduled-tasks.md, index.md, current-state.md and log.md mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
[2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
Window audited: 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and produced no entry, so this run covers two days.
Pages written
incidents/2026-10-09-cron-provider-timeout-storm.md— new.incidents/2026-10-09-memory-recall-relevance.md— new.systems/scheduled-tasks.md— Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped.systems/mnemosyne.md— live tier counts refreshed; new dated counts line with a pointer to the recall incident.current-state.md— Recent Changes (2026-10-09) block; frontmatter dates bumped.index.md— last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added.systems/skills-index.md+raw/stocktake/freshness-20261008.txt— the stranded 2026-10-08 03:01 stocktake artifacts committed.
Findings worth the ink
- The fleet had its worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight remain open. The watchdog (96eb95ce12f2) failed twice as a cascade — it exits 1 when it finds ERROR lines inagent.log, i.e. it reported the storm rather than suffering it. - The 2026-10-08 window was never audited — this job's own 23:59 run failed with the same timeout (incident
cbfe43_fa81052baaf7) and committed nothing. - The stocktake has a two-day gap. 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no
freshness-20261009.txtexists.1a8e90ea2271is the one job still atfailure_streak: 2. - An instruction was reported done and was not. Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that
62619aa6bcb3was "stopped and deleted". It is still enabled (paused_at: null) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit. - Memory recall relevance — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is not in live config (
memory.mnemosynecarries no recall-weight keys; no top-levelmnemosyne:key).
Vault mirror: the touched pages mirrored into /home/hermes/wiki/agent-estate-wiki/ for the iPad/Ignis view.
No secrets written; credential-shaped values referenced by job ID and connection ID only.
2026-10-10 — Stocktake (manual run, wiki-maintain.sh)
systems/skills-index.mdregenerated — 346 skills / 70 categories (unchanged count; frontmatterupdated/verified_onrefreshed to 2026-10-10). Live on-disk count is 347SKILL.md; the one-skill gap is the known de-dup ofproductivity/obsidian-git-sync/obsidian-git-sync/.- Freshness audit →
raw/stocktake/freshness-20261010.txt: 62 stale (≥21d), 23 fresh, 7 no-date of 92 pages. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. Nofreshness-20261009.txtexists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands. current-state.md— new 2026-10-10 section. The 2026-10-09 timeout storm window re-tallied fromexecutions.db(2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163completed, 15failed— the prior audit recorded 162/15+1 running; the strayrunningrow is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page).- DHL monitor re-probed, still enabled.
62619aa6bcb3—enabled: True,paused_at: null,0 8,11,14,17,20 * * *,last_status: ok. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. No cron mutation performed by this audit — retirement stays open item #1. infrastructure/tailscale.mdverified againsttailscale serve status— all 13 tailnet Serve routes plus the:8443/webhookFunnel match the page. No drift; not edited.- Not verified / left alone: the Quartz rebuild step (no
quartz.config.ts,npxcannot fetchquartz—public/was NOT refreshed, confirming the carried open item);docker ps(permission denied on/var/run/docker.sockfrom this session, soinfrastructure/docker-services.mdcould not be probed and is left stale); the 80d2026-07-22cluster (no live source to check against). - Sessions in the 2026-10-09 window (
state.db, epoch REAL): 19 total — 6cron, 11ios, 1talaria-ios, 1telegram. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — thedeliveredcount includes failure notices).
No secrets written; job/incident IDs only.
[2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-11T00:19). Day tally: 10-07 177/1,
10-08 178/2, 10-09 163/15, 10-10 174 completed / 0 failed. Non-clean last_status 10 -> 5.
Incidents: 7 closed during 10-10, 5 open (3 jobs). Open item #1 closed: the DHL monitor
62619aa6bcb3 is retired and verified absent from jobs.json by ID lookup (the 10-07 and 10-09
"deleted" claims were false).
Runtime/memory. The 2026-09-28 managed-runtime-dependency class recurred in the gateway env:
mnemosyne (+native deps) and elevenlabs were missing from the committed uv env
795020dbf24343cd9c52df24723dca83 that activate_dependencies() loads. Repaired 21:26-21:34 and
verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated;
repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open.
New page. systems/paseo - Paseo daemon v0.11.2 on 0.0.0.0:6767 (user unit
paseo-daemon.service); recorded the daemon run vs daemon start supervisor trap, the
/usr/bin/paseo 0.7.2 vs daemon 0.11.2 version split, and the truncated-cline false provider fault.
Platform. Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 answers
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined.
Pages touched: systems/paseo.md (new), systems/scheduled-tasks.md, systems/hermes-agent.md,
systems/mnemosyne.md, systems/messaging-integrations.md,
incidents/2026-09-28-hermes-managed-runtime-deps.md, current-state.md, index.md, log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.
[2026-10-11] stocktake | Daily maintenance + live drift corrections
- Ran
wiki-maintain.sh: skills-index regenerated (346 skills / 70 categories), freshness audit written toraw/stocktake/freshness-20261011.txt(62 stale / 24 fresh / 7 no-date of 93 pages) - OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed. Verified via
pct list,pct config,ssandtailscale status: CT600omniroute(100.88.81.19) is RUNNING and serving:20128; CT601omni(100.93.204.4) is STOPPED withonboot=0. Hermes resolvescustom:omni→omniroute:20128→ CT600, not CT601. systems/omniroute.md rewritten with the live node table. - Third OmniRoute node found:
omniroute-1at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provideromni-101(5384 models, defaultsmart-route, carriesskills_hub). Not inpct list— identity unconfirmed. - Ignis duplicate found: a second
ignis-ignis-1container runs on CT200 (imageignis-server:latest, host port 8092, vault/opt/ignis/vaults), separate from CT460'signis(port 8080, vault/home/hermes/wiki/agent-estate-wiki). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them. - Tailscale route table corrected against live
tailscale serve status:/agent-estatetarget is barehttp://127.0.0.1:8769(not.../8769/agent-estate-role-dashboard.html); previously-missing tailnet-only:9443→127.0.0.1:5173documented. CT201 routes refreshed (8085 targets +/steel5173 +/steel-api3000). - Proxmox inventory rebuilt from live
pct list+qm list: added CT475agentzeroand VM110omarchy; corrected 282ubuntu-server-xfceto stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). VM403open-clawand VM401hermes-oldare absent fromqm list— flagged unconfirmed rather than guessed. - Nanobot verified on CT333: gateway
nanobot-gateway(PID 2259048) listening on 8765/8766; NOT a systemdnanobot.service/gateway.service(both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified. - Docker services page corrected: the old "Docker runtime not available" note was a CT460 socket-permission artifact (
/var/run/docker.sockisroot:docker srw-rw----,hermesuser not in groupdocker, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo). - Flagged for follow-up, root cause not investigated:
mycompose-db-1on CT200 is inRestarting (1)— crash-looping. - Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d.
- Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten.
[2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
Cron. 29 jobs, 26 enabled / 3 disabled (jobs.json 2026-10-12T00:10). Day tally: 10-07 177/1,
10-08 178/2, 10-09 163/15, 10-10 175 completed / 0 failed (finalised), 10-11 171 completed / 0 failed + 1 running — two consecutive clean days. Non-clean last_status steady at 5;
0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected).
Corrected a false alarm instead of logging a fault. The 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION ("capture stalled 10.7 h") was a naive-UTC vs AWST comparison error — real gap 2.7 h, the row it cited landed inside the session that wrote it. Recorded in systems/mnemosyne and systems/scheduled-tasks so the next run of that monitor is not triaged as a capture outage. Live counts refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %.
New content. queries/retained-research-answers was a 2026-07-22 stub and is now a real page — the
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB;
-CompressLevel 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
was undocumented; the live datastore table plus host mount points are now in infrastructure/proxmox
(/mnt/ssd0 93 % / ~63 GiB free is the pressure point).
Stale labels fixed. incidents/2026-08-18-mnemosyne-capture-pipeline carried status: active from
August despite capture being verifiably healthy — now superseded, with the live evidence and a caution about
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
and the skills-index blurb no longer claims no regeneration since 2026-10-07.
Verified, not changed. The blocked-page-recovery lesson that session reported patching
(recovered: false = archive routes failed, not task failed; API-first rung) is present in the live skill.
Pages touched: systems/scheduled-tasks.md, systems/mnemosyne.md, infrastructure/proxmox.md,
queries/retained-research-answers.md, incidents/2026-08-18-mnemosyne-capture-pipeline.md,
current-state.md, index.md, log.md.
No secrets written; job/incident IDs, paths and tailnet addresses only.