Files
agent-estate-wiki/log.md
T
Tony0410 0c4ed3350b wiki audit 2026-10-02: 2 new cron failures, cliprox/terra connection auth incident, delivery-failure semantics corrected, profile rewrite, broken wikilinks repaired
- new: incidents/2026-10-01-cliproxy-connection-auth-expired.md (HTTP 401, credential fault, cannot self-recover)
- systems/scheduled-tasks.md: 30 jobs / 27 enabled; failing 3 -> 6; streaks advanced; wiki-tree drift recorded
- systems/messaging-integrations.md: two distinct thread-failure variants (lost vs thread-less fallback)
- systems/current-profile.md: full rewrite from live config (no profiles/default/, smart-route, OmniRoute)
- current-state.md / index.md / log.md: 2026-10-02 entry
- repair: copied 5 vault-only pages referenced by index.md (ignis, nextcloud, 3 incidents) so wikilinks resolve
- include pending stocktake output: skills-index.md 2026-09-27, raw/stocktake/freshness-2026-09-2{6,7}.txt
2026-10-02 00:17:15 +08:00

37 KiB
Raw Blame History

Wiki Log

Chronological record of wiki actions. Append-only.

[2026-07-22] init | Fresh Hermes Systems Wiki

  • Created technical wiki at /home/hermes/wiki
  • Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
  • Wrote SCHEMA.md with 11 operating rules
  • Wrote index.md as curated Start here dashboard
  • Reseeded system/infrastructure/runbook stubs for existing homelab
  • Git initialized, local-only
  • No secrets, passwords, tokens, or private keys written to vault

[2026-07-22] audit | External audit pass (Claude, at Anthony's request)

  • Merged duplicate pages: systems/omniRoute.md (empty stub) -> kept systems/omniroute.md (populated); repointed 5 links
  • Merged duplicate pages: runbooks/safely-update-hermes.md -> kept as runbooks/update-hermes-safely.md (fuller content); repointed 2 links
  • Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
  • Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
  • Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
  • Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
  • Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts

[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)

  • Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
  • Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
  • Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
  • Fixed a broken review_after: 0001-01-01 in decisions/keep-openrouter-as-fallback.md
  • New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
  • queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate

[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)

  • Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
  • Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
  • Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
  • Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
  • Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
  • Confirmed: Mnemosyne is, and has only ever been, this box's memory backend

[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)

  • Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
  • These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki

[2026-07-25] operations | Tool Search and gateway resilience

  • Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
  • Documented how new skill categories are derived from the first folder below ~/.hermes/skills/
  • Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
  • Recorded the I/O-controller delegation limitation so an ineffective IOWeight is not mistaken for active protection
  • Added health checks, exact effective settings, backup location, and rollback procedure
  • No secrets, tokens, or private credentials were written to the wiki

[2026-07-28] dashboard | Jobs Dashboard feature additions

  • Fixed bulk status update (payload shape mismatch: {ids, status} → {ids, updates: {status}})
  • Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
  • Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
  • Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
  • Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion Notes / JD Summary
  • Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
  • Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
  • Updated Synergy and Dept of Communities to Phone Screen status
  • Updated wiki runbook with full dashboard feature documentation and API endpoint table

[2026-07-27] incident | Job Radar search and dashboard recovery

  • Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
  • Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require MISSING_COVER_LETTER_COUNT=0
  • Recorded the production cron model (gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5)
  • Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
  • Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both :443 Serve and :8443/webhook Funnel
  • Added runbooks/job-radar and updated scheduled-task, Tailscale and index references
  • Verified no secrets, API keys or private credentials were written to the wiki

[2026-07-29] update | Steel Browser deployment — browser backend switch

  • New page: systems/browser-backend.md — rewritten to reflect Steel Browser as primary, Camoufox as fallback
  • New page: comparisons/steel-vs-camoufox.md — comparison of the two backends
  • Updated: infrastructure/hosts.md — added CT201 and CT460 to verified hosts table
  • Updated: infrastructure/docker-services.md — added Steel Browser compose file
  • Updated: infrastructure/tailscale.md — added CT201 Steel serve routes
  • Updated: index.md — browser-backend link description
  • Updated: current-state.md — added 2026-07-29 changes section
  • Config change: browser.cdp_url set to http://100.96.244.39:9223 in ops profile

[2026-07-30] update | Open WebUI updated on CT245

  • Updated image to ghcr.io/open-webui/open-webui:main (new digest e97bf9531916, Jul 27)
  • Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
  • Restarted container after OOM hang
  • Updated infrastructure/proxmox.md

[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed

  • Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
  • Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
  • Ran mnemosyne reindex --yes to rebuild all embeddings — working memory now at 100%
  • Updated health check script (mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py)
  • Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
  • Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
  • DB grew from 8.7M to 21M with the new embeddings

[2026-08-01] incident | Mnemosyne capture pipeline repaired

  • Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no working_memory row existed.
  • Corrected stringified structured YAML (sync_roles, skip_contexts, reflect, sleep_threshold, ignore_patterns) and enabled profile isolation; hermes config check passed.
  • Verified the repair with a second CLI marker that persisted as an exact normalized [USER] database row.
  • Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route :9119; added authoritative identity/preference/workflow slots.
  • Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.
  • Added incidents/2026-08-01-mnemosyne-capture-failure and refreshed systems/mnemosyne plus systems/memory-backend.

[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation

[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync

  • Ops profile (~/.hermes/profiles/ops/config.yaml): enabled mnemosyne plugin with profile_isolation: true, allow_tool_override: false, shared_surface_read: false
  • Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
  • Verified end-to-end: memory writes land correctly in both profiles independently
  • Updated systems/mnemosyne — profile isolation section now documents ops config
  • Updated current-state — added ops profile Mnemosyne entry
  • Updated runbooks/multiplexer-setup — fixed "Elephant ops" → "Tusk ops" in architecture diagram
  • Fixed systems/memory-backend — "Shared surface empty" section renamed to reflect intentional isolation decision

[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation

  • Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances
  • Reworked deepseek-flash-free and good-free on CT600: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort
  • Created incidents/2026-08-04-omniroute-free-deepseek-silent-stops
  • Updated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes
  • Updated systems/omniroute — added Recent Changes (2026-08-04) section
  • Updated systems/model-providers — added mitigation entry under Notable Model Updates, updated updated date

[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite

  • OmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. is_active=0 does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.
  • Updated systems/omniroute — added Recent Changes (2026-08-07) section with full root cause analysis
  • Updated systems/nanobot — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.
  • Updated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
  • Rewrote systems/scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.
  • Wiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
  • Updated index.md — last-updated date 2026-08-04 → 2026-08-07

[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review

  • Portainer password reset documented: CT245 admin user confirmed as anthony (not admin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state.
  • SFTPGo security review documented: CT299 findings — root access, unpinned fingerprint, // root path, F1 quota mystery. Added to current-state and infrastructure/proxmox.

[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review

  • Cron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker from freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named jobs (e.g. vertex/gemini-2.5-flash, deepseek-v4-flash-free) → smart-route. Daily Marketing Job Radar → gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run (ID a9359b7db27b) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, CT333 Nanobot has live jobs.json with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.
  • Updated current-state — added 2026-08-08 changes section
  • Updated systems/scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues

[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue

  • Cron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:
    • TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
    • RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
  • Root cause identified: Disk at 83% (11GB free on 65GB), state.db at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.
  • Updated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
  • Updated systems/scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10
  • OpenCode WebSocket proxy errors noted: Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient.
  • No new pages created — all changes applied to existing pages

[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals

  • OmniRoute AutoRoute variants documented: auto/best-free (Auto Cheap) and auto/coding-free are virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: auto/best-free routed to Antigravity. Updated systems/omniroute.
  • OpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains memory.provider: mnemosyne. Added to comparisons/memory-backends.
  • FreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default freellmapi-auto. Added to comparisons/model-routers.
  • No new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.

[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration

  • OmniRoute migrated from CT600 to CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list, grep, Tailscale DNS).
  • Updated systems/omniroute — Current Configuration rewritten for omni/CT601, added Recent Changes (2026-08-13), bumped updated/verified_on to 2026-08-13.
  • Updated infrastructure/proxmox — CT600 marked stopped/retired, CT601 added running; bumped updated.
  • Updated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.
  • Updated infrastructure/hosts — OmniRoute proxy entry → omni; CT600 reference marked verified/retired.
  • Updated index — last-updated date 2026-08-10 → 2026-08-13.
  • No new pages created; all changes applied to existing pages. No secrets written.

[2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated

  • Skills index drift incident: systems/skills-index.md claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating regenerate-skills-index.sh script, and adding cron job 1a8e90ea2271 (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See incidents/2026-08-15-skills-index-drift.
  • Updated systems/skills-index — added auto-regeneration notice, updated counts (66→330), bumped updated/verified_on to 2026-08-15.
  • Updated systems/scheduled-tasks — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates.
  • Updated systems/hermes-agent — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates.
  • Updated index.md — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15.
  • Created incidents/2026-08-15-skills-index-drift — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron).
  • No secrets written.

[2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash

  • New page: systems/nextcloud.md — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages.
  • Updated current-state — added Recent Changes (2026-08-17) section with ncdownloader fix details.
  • Updated infrastructure/proxmox — CT270 entry now shows Nextcloud version and ncdownloader status.
  • Updated index.md — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17.
  • No secrets written.

[2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk

  • OmniRoute upstream capacity pressure documented: ServiceUnavailableError: Structurally heavy chat request capacity is busy reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded.
  • Agent Zero sluggishness confirmed as orchestration overhead: GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026).
  • Hermes model verification quirk documented: Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config http://omni:20128/v1 is correct — /v1 is required.
  • Updated systems/omniroute — added Recent Changes (2026-08-16) section, bumped updated/verified_on to 2026-08-16.
  • Updated current-state — added Recent Changes (2026-08-16) section, bumped updated to 2026-08-16.
  • Updated index.md — bumped last-updated date to 2026-08-16.
  • No new pages created. No secrets written.

[2026-08-18] incident | Mnemosyne capture pipeline broken (active)

  • New page: incidents/2026-08-18-mnemosyne-capture-pipeline — working memory capture (sync_turn) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in run_agent.py:_sync_external_memory_for_turn(). MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification.
  • Updated systems/mnemosyne — added 2026-08-18 incident to History and Evidence, bumped updated/verified_on to 2026-08-18, added cross-reference to Related.
  • Updated current-state — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped updated to 2026-08-18.
  • Updated index.md — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18.
  • Marked incidents/2026-08-01-mnemosyne-capture-failure — status changed from active to resolved (that earlier incident was fixed).
  • No secrets written.

[2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history

  • Cron job failures documented (Weekly Review session): Three distinct failure patterns identified in past 7 days:
    • wiki-stocktake-maintenance (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from cx/gpt-5.6-luna → go-cheap/opencode-go/gpt-5.6-luna, job unpinned. Requires explicit provider/model pinning.
    • CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected.
    • Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping.
  • Updated systems/scheduled-tasks — added Known Issues section with cron failure patterns, bumped updated/verified_on to 2026-08-23.
  • Updated systems/mnemosyne — added Mnemosyne Watchdog Aug 19 incident to History section, bumped updated/verified_on to 2026-08-23.
  • Updated current-state — added Recent Changes (2026-08-23) section with cron failures summary.
  • Updated index.md — bumped last-updated date to 2026-08-23.
  • No secrets written.

[2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status

  • New page: systems/ignis.md — documented Ignis browser-native Obsidian (Docker ignis-subpath:0.8.9, port 8080, Tailscale Serve at hermes.kangaroo-eel.ts.net/ignis). Volume mount: /home/hermes/wiki/agent-estate-wiki:/vaults/wiki. Verified live: container running, Tailscale route active.
  • Updated systems/model-providers — added command-code provider (https://api.commandcode.ai/provider/v1) to Current Provider Stack. Bumped updated/verified_on to 2026-08-19.
  • Updated systems/omniroute — added Recent Changes (2026-08-19) section documenting Command Code provider, deepseek-flash-mix combo, and Tavili search proxy status. Bumped updated/verified_on to 2026-08-19.
  • Updated infrastructure/docker-services — added Ignis compose file and service definition.
  • Updated infrastructure/hosts — added Ignis to CT460 row.
  • Updated systems/obsidian — added Ignis as browser interface, corrected vault path to agent-estate-wiki/, added Quartz static site reference.
  • Updated current-state — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status.
  • Updated index.md — added Ignis link to Core systems, bumped last-updated to 2026-08-19.
  • No secrets written.

[2026-08-26] update | Nightly wiki audit — Cron job reconciliation

  • Updated systems/scheduled-tasks — reconciled live hermes cron list (27 jobs) against wiki table (29 entries). Removed: Workday start reminder (no longer in live list), CT 221 Miner Watchdog (job d5516d0a5147 removed; CT221 status unknown), Review Agent Estate Phase One (one-time job completed 2026-08-08). Added: weekly-save-restricted-bot-restart (0 3 * * 0, local, no-agent). Updated count from 29 to 27.
  • Updated current-state — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence.
  • Updated index.md — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27).
  • Known Issues cleanup — removed stale CT221 miner watchdog SSH failure entry (job no longer exists).
  • No secrets written.

[2026-09-02] incident | SFTPGo outage on CT299

  • New page: incidents/2026-09-02-sftpgo-outage.md — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), tailscaled died; PVE pct start failed due to 5 stale lxc-attach processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop 299 --skiplock + pct start 299. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
  • Updated current-state — added Recent Changes (2026-09-02) section, bumped updated/verified_on to 2026-09-04.
  • Updated infrastructure/proxmox — CT299 entry already documented incident; no change needed.
  • No secrets written.

[2026-08-28] update | Department of Communities role and social-media audit

  • New page: reports/communities-wa-social-media-audit-2026 — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role.
  • Updated systems/current-profile — recorded current employment context and linked the report.
  • Updated index.md — added the report to the Reports section and refreshed the last-updated date.
  • No secrets written.

[2026-09-20] update | Nightly wiki audit — No changes in past 24h

  • No new pages created — no new systems, decisions, or incidents in the past 24 hours.
  • No existing pages updated — all wiki content remains current.
  • Freshness audit run: freshness-20260920.txt generated at 03:01 AWST (scheduled cron).
  • Skills index auto-regenerated: systems/skills-index.md updated at 03:01 AWST (scheduled cron 1a8e90ea2271).
  • Scheduled tasks reconciled: systems/scheduled-tasks.md updated at 03:02 AWST — 26 active jobs verified from live hermes cron list.
  • No secrets written.

[2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation

  • New page incidents/2026-09-28-hermes-managed-runtime-deps — the 2026-09-28 → 30 cron-fleet outage: jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1), caused by ModuleNotFoundError: No module named 'dotenv' / 'mnemosyne' in the uv-managed runtime venvs under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (all 10 venvs, pip install --target), run every 15 min by job 96eb95ce12f2.
  • Updated systems/scheduled-tasks.md — table rebuilt from live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to d40085da631c, f77b7e46bd09, 0a7f933d06e2. Known Issues refreshed: job-radar 403 MODEL_NOT_IN_PLAN (streak 5), Telegram thread-delivery failures, weekly-save-restricted-bot-restart script restored, residual pre-fix dispatch errors on cbfe434a3b5e/1a8e90ea2271.
  • Updated systems/hermes-agent.md — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model custom → http://100.78.63.20:20128 / smart-route; skills 330 → 341; cron count 29 → 30.
  • Updated current-state.md — _config_version: 46 and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section.
  • Updated runbooks/job-radar.md — corrected primary model row (custom:omni-101 / gpt-5.6-terra via command-code) and added the 403 MODEL_NOT_IN_PLAN failure mode with recovery steps.
  • Updated systems/messaging-integrations.md — Telegram allowed_chats/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup).
  • Updated index.md — last-updated to 2026-09-30, cron count to 30, new incident listed.
  • No secrets written.

[2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene

  • No new pages created — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages.
  • Updated systems/scheduled-tasks.md — table and counts re-verified against live ~/.hermes/cron/jobs.json: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — dca8482e4f76 (403 MODEL_NOT_IN_PLAN, ×5), 1a8e90ea2271 (HTTP 410, model retired upstream, ×7), fad5a86979d4 (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: 1b23c042e17d and c8d52904e1d6 delivery targets now telegram:-1004321904721:102; c0a9fa314542 flagged for a revoked Telegram bot token; bare-telegram targets noted on 0e753e9720fd, fad5a86979d4, 066bb5191280. Known Issues restructured into Resolved / Open / Historical.
  • Updated systems/messaging-integrations.md — new Delivery target semantics section: a bare deliver: "telegram" resolves to the home DM (1793951355), not to the primary group, so a group topic ID attached to it hard-fails with Message thread not found and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs.
  • Updated systems/model-providers.md — provider stack rebuilt from live config.yaml: 5 custom providers (omni-101 5,384 models, Omniroute:20128 5,398, litellm 1,728, command-code 71, guacano 46); custom:omni recorded as removed. Default model corrected resilient-free → smart-route. New Retired / Unavailable Models table (retired qwen3.5:397b, plan-gated gpt-5.6-terra, catalog-missing deepseek-v4-flash) with the lesson that a raw model pin never self-recovers.
  • Updated systems/mnemosyne.md — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as [USER] [Anthony Martin] memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy.
  • Updated systems/hermes-agent.md — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins.
  • Updated index.md and current-state.md — last-updated bumped to 2026-10-01 with a Recent Changes block.
  • Vault sync: the same page edits were mirrored into /home/hermes/wiki/agent-estate-wiki/wiki/ (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under wiki/, not at the tree root, and index.md sits at the vault root.
  • No secrets written.

[2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split

  • New page: incidents/2026-10-01-cliproxy-connection-auth-expired.md — job 06a0a1c8704f (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard (provider custom:Omniroute:20128, model cliprox/terra). Documented as its own incident because it is a credential fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (1b23c042e17d, 79f9bb41ecf5).
  • Updated systems/scheduled-tasks.md — counts re-verified against live ~/.hermes/cron/jobs.json: still 30 jobs (27 enabled, 3 disabled), but the failing set moved 3 → 6. New on 2026-10-01: 06a0a1c8704f (HTTP 401), 0e753e9720fd (HTTP 400 — same NVIDIA model pin as the already-failing fad5a86979d4, confirming the shared-pin diagnosis), 066bb5191280 (delivery failed; report itself generated). Streaks advanced: dca8482e4f76 ×5→×6, 1a8e90ea2271 ×7→×8, fad5a86979d4 ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees.
  • Updated systems/messaging-integrations.md — corrected a wrong page fact. The page previously stated that a bad thread_id always loses the message. The 2026-10-01 records show two variants: Message thread not found (hard failure, message lost — the 2026-09-30 case) and configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id (a thread-less fallback send does occur, yet the run is still recorded delivery_outcome: failed). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-telegram jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today".
  • Updated systems/current-profile.md — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no ~/.hermes/profiles/default/ directory (the default profile is the top-level config; ops is the only named sub-profile), the model default is smart-route (not freellmapi-auto), the default provider is custom → http://100.78.63.20:20128 / OmniRoute (litellm is now only the last fallback entry), and the config is _config_version: 46. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings.
  • Updated current-state.md — new Recent Changes (2026-10-02) block; frontmatter dates bumped.
  • Updated index.md — last-updated to 2026-10-02; cron line now notes the failing count.
  • Structural finding — the wiki exists as two trees and they have drifted both ways. /home/hermes/workspace/agent-estate-wiki is canonical: it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy the iPad reads (pages under wiki/, no git remote). The audit prompt names only the vault path. The trees have also lost pages in opposite directions — systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, agents/, incidents/2026-09-02-sftpgo-outage.md and reports/communities-wa-social-media-audit-2026.md exist only in canonical — so canonical index.md currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to both trees.
  • No secrets written; all credential-shaped values in config remain redacted.