1867 lines
303 KiB
Markdown
1867 lines
303 KiB
Markdown
# Wiki Log
|
||
|
||
> Chronological record of wiki actions. Append-only. ## [2026-07-22] init | Fresh Hermes Systems Wiki
|
||
- Created technical wiki at `/home/hermes/wiki`
|
||
- Structure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/
|
||
- Wrote SCHEMA.md with 11 operating rules
|
||
- Wrote index.md as curated Start here dashboard
|
||
- Reseeded system/infrastructure/runbook stubs for existing homelab
|
||
- Git initialized, local-only
|
||
- No secrets, passwords, tokens, or private keys written to vault
|
||
|
||
## [2026-07-22] audit | External audit pass (Claude, at Anthony's request)
|
||
- Merged duplicate pages: `systems/omniRoute.md` (empty stub) -> kept `systems/omniroute.md` (populated); repointed 5 links
|
||
- Merged duplicate pages: `runbooks/safely-update-hermes.md` -> kept as `runbooks/update-hermes-safely.md` (fuller content); repointed 2 links
|
||
- Added missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)
|
||
- Flagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin
|
||
- Linked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they're reachable from Start Here
|
||
- Verified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)
|
||
- Did not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts
|
||
|
||
## [2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony's request)
|
||
- Populated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory
|
||
- Corrected systems/nanobot.md: Nanobot is actually running, on LXC 333 ("clawtest"), not this host — earlier "not running" claim was true only for ct460 and got conflated with pve's own IP
|
||
- Found and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten
|
||
- Fixed a broken `review_after: 0001-01-01` in decisions/keep-openrouter-as-fallback.md
|
||
- New security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)
|
||
- queries/retained-research-answers.md intentionally left as a stub — it's a running research log to be filled by Hermes over time, not something system inspection can populate
|
||
|
||
## [2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)
|
||
- Anthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring
|
||
- Deleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md
|
||
- Rewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box's predecessor backend
|
||
- Removed Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md
|
||
- Left in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use
|
||
- Confirmed: Mnemosyne is, and has only ever been, this box's memory backend
|
||
|
||
## [2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)
|
||
- Deleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived
|
||
- These were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki
|
||
|
||
## [2026-07-25] operations | Tool Search and gateway resilience
|
||
- Documented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup
|
||
- Documented how new skill categories are derived from the first folder below `~/.hermes/skills/`
|
||
- Documented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI
|
||
- Recorded the I/O-controller delegation limitation so an ineffective `IOWeight` is not mistaken for active protection
|
||
- Added health checks, exact effective settings, backup location, and rollback procedure
|
||
- No secrets, tokens, or private credentials were written to the wiki
|
||
|
||
## [2026-07-28] dashboard | Jobs Dashboard feature additions
|
||
- Fixed bulk status update (payload shape mismatch: `{ids, status}` → `{ids, updates: {status}}`)
|
||
- Fixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)
|
||
- Added Phone Screen as a dedicated KPI filter card (separated from Interviews)
|
||
- Added Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)
|
||
- Added rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion `Notes / JD Summary`
|
||
- Added document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)
|
||
- Added 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager
|
||
- Updated Synergy and Dept of Communities to Phone Screen status
|
||
- Updated wiki runbook with full dashboard feature documentation and API endpoint table
|
||
|
||
## [2026-07-27] incident | Job Radar search and dashboard recovery
|
||
- Documented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow
|
||
- Added the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require `MISSING_COVER_LETTER_COUNT=0`
|
||
- Recorded the production cron model (`gpt-5.6-terra`) and verified first inherited fallback (`opencode-go/mimo-v2.5`)
|
||
- Documented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing
|
||
- Recorded the CT460 Tailscale route-table loss and corrected atomic restoration of both `:443` Serve and `:8443/webhook` Funnel
|
||
- Added [[runbooks/job-radar]] and updated scheduled-task, Tailscale and index references
|
||
- Verified no secrets, API keys or private credentials were written to the wiki
|
||
|
||
## [2026-07-29] update | Steel Browser deployment — browser backend switch
|
||
- **New page:** `systems/browser-backend.md` — rewritten to reflect Steel Browser as primary, Camoufox as fallback
|
||
- **New page:** `comparisons/steel-vs-camoufox.md` — comparison of the two backends
|
||
- **Updated:** `infrastructure/hosts.md` — added CT201 and CT460 to verified hosts table
|
||
- **Updated:** `infrastructure/docker-services.md` — added Steel Browser compose file
|
||
- **Updated:** `infrastructure/tailscale.md` — added CT201 Steel serve routes
|
||
- **Updated:** `index.md` — browser-backend link description
|
||
- **Updated:** `current-state.md` — added 2026-07-29 changes section
|
||
- **Config change:** `browser.cdp_url` set to `http://100.96.244.39:9223` in ops profile
|
||
|
||
## [2026-07-30] update | Open WebUI updated on CT245
|
||
- Updated image to `ghcr.io/open-webui/open-webui:main` (new digest e97bf9531916, Jul 27)
|
||
- Bumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image
|
||
- Restarted container after OOM hang
|
||
- Updated `infrastructure/proxmox.md`
|
||
|
||
## [2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed
|
||
- Health monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap
|
||
- Working memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded
|
||
- Ran `mnemosyne reindex --yes` to rebuild all embeddings — working memory now at 100%
|
||
- Updated health check script (`mnemosyne-health-check.sh`) to report actual embedding coverage via new helper script (`check_embedding_coverage.py`)
|
||
- Created embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%
|
||
- Added auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed
|
||
- DB grew from 8.7M to 21M with the new embeddings
|
||
|
||
## [2026-08-01] incident | Mnemosyne capture pipeline repaired
|
||
- Confirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no `working_memory` row existed. - Corrected stringified structured YAML (`sync_roles`, `skip_contexts`, `reflect`, `sleep_threshold`, `ignore_patterns`) and enabled profile isolation; `hermes config check` passed. - Verified the repair with a second CLI marker that persisted as an exact normalized `[USER]` database row. - Corrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route `:9119`; added authoritative identity/preference/workflow slots. - Reduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained. - Added [[incidents/2026-08-01-mnemosyne-capture-failure]] and refreshed [[systems/mnemosyne]] plus [[systems/memory-backend]]. ## [2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation
|
||
- Rewrote [[comparisons/steel-vs-camoufox]] — full Steel experiment story, timeline, lessons learned
|
||
- Rewrote [[systems/browser-backend]] — Camoufox primary, Steel disabled
|
||
- Updated [[current-state]] — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename
|
||
- Updated [[concepts/browser-automation]] — added Steel experiment lesson
|
||
- Created [[decisions/rollback-steel-to-camofox]] — formal decision record
|
||
- Updated [[systems/scheduled-tasks]] — job count 25, Uber Eats upgrade, GH Trending Daily removed
|
||
- Updated [[systems/mnemosyne]] — vector embedding coverage gap documented
|
||
- Updated [[systems/model-providers]] — DeepSeek V4 Flash 0731 details
|
||
- Created [[reports/uber-eats/index]] — landing page for monthly spend reports
|
||
- Updated [[decisions/index]] — linked new decision page
|
||
- Updated [[index.md]] — browser description, reports section, last-updated date
|
||
- Created nightly wiki audit cron job (23:59 AWST)
|
||
|
||
## [2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync
|
||
- Ops profile (`~/.hermes/profiles/ops/config.yaml`): enabled `mnemosyne` plugin with `profile_isolation: true`, `allow_tool_override: false`, `shared_surface_read: false`
|
||
- Tusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination
|
||
- Verified end-to-end: memory writes land correctly in both profiles independently
|
||
- Updated [[systems/mnemosyne]] — profile isolation section now documents ops config
|
||
- Updated [[current-state]] — added ops profile Mnemosyne entry
|
||
- Updated [[runbooks/multiplexer-setup]] — fixed "Elephant ops" → "Tusk ops" in architecture diagram
|
||
- Fixed [[systems/memory-backend]] — "Shared surface empty" section renamed to reflect intentional isolation decision
|
||
|
||
## [2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation
|
||
- Incident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances
|
||
- Reworked `deepseek-flash-free` and `good-free` on CT600: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort
|
||
- Created [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]]
|
||
- Updated [[current-state]] — added OmniRoute free-DeepSeek mitigation to Recent Changes
|
||
- Updated [[systems/omniroute]] — added Recent Changes (2026-08-04) section
|
||
- Updated [[systems/model-providers]] — added mitigation entry under Notable Model Updates, updated `updated` date
|
||
|
||
## [2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite
|
||
- **OmniRoute three-layer fix documented:** npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. `is_active=0` does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-07) section with full root cause analysis
|
||
- **Updated [[systems/nanobot]]** — status changed from "active" to "degraded" (DOWN, connection refused on 8900). Updated confidence, sources, verified_on. - **Updated [[current-state]]** — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump
|
||
- **Rewrote [[systems/scheduled-tasks]]** — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28. - **Wiki audit cron error noted:** model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job
|
||
- **Updated [[index.md]]** — last-updated date 2026-08-04 → 2026-08-07
|
||
|
||
## [2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review
|
||
- **Portainer password reset documented:** CT245 admin user confirmed as `anthony` (not `admin`). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to [[current-state]]. - **SFTPGo security review documented:** CT299 findings — root access, unpinned fingerprint, `//` root path, F1 quota mystery. Added to [[current-state]] and [[infrastructure/proxmox]]. ## [2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review
|
||
- **Cron job model repinning:** 13 jobs repinned from dead/stale provider combos. 3 from `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker from `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named jobs (e.g. `vertex/gemini-2.5-flash`, `deepseek-v4-flash-free`) → `smart-route`. Daily Marketing Job Radar → `gpt-5.6-terra`. - **Hermes UI vite config revert:** Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460. - **Phase One estate review completed:** Manual cron run (ID `a9359b7db27b`) verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, CT333 Nanobot has live `jobs.json` with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes. - **Updated [[current-state]]** — added 2026-08-08 changes section
|
||
- **Updated [[systems/scheduled-tasks]]** — Phase One review marked completed, model repinning noted in Known Issues
|
||
|
||
## [2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue
|
||
- **Cron job failures documented:** Four jobs failed on 2026-08-10 with two error patterns:
|
||
- TimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest
|
||
- RuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup
|
||
- **Root cause identified:** Disk at 83% (11GB free on 65GB), `state.db` at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues. - **Updated [[current-state]]** — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors
|
||
- **Updated [[systems/scheduled-tasks]]** — added session storage failures to Known Issues, updated verified_on to 2026-08-10
|
||
- **OpenCode WebSocket proxy errors noted:** Repeated "OpenCode service unavailable" in hermes node journal (Aug 11 00:08). Process running, likely transient. - **No new pages created** — all changes applied to existing pages
|
||
|
||
## [2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals
|
||
- **OmniRoute AutoRoute variants documented:** `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: `auto/best-free` routed to Antigravity. Updated [[systems/omniroute]]. - **OpenViking memory provider evaluated, NOT adopted:** single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains `memory.provider: mnemosyne`. Added to [[comparisons/memory-backends]]. - **FreeLLMAPI router internals documented:** 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes's default `freellmapi-auto`. Added to [[comparisons/model-routers]]. - **No new pages created** — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents. ## [2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration
|
||
- **OmniRoute migrated from CT600 to CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute.kangaroo-eel.ts.net` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Pre-migration backup at `~/.hermes/config.yaml.bak.omniroute-removal`. All facts verified live (`pct list`, `grep`, Tailscale DNS). - **Updated [[systems/omniroute]]** — Current Configuration rewritten for `omni`/CT601, added Recent Changes (2026-08-13), bumped `updated`/`verified_on` to 2026-08-13. - **Updated [[infrastructure/proxmox]]** — CT600 marked stopped/retired, CT601 added running; bumped `updated`. - **Updated [[current-state]]** — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix. - **Updated [[infrastructure/hosts]]** — OmniRoute proxy entry → `omni`; CT600 reference marked verified/retired. - **Updated [[index]]** — last-updated date 2026-08-10 → 2026-08-13. - No new pages created; all changes applied to existing pages. No secrets written. ## [2026-08-15] update | Nightly wiki audit — Skills index drift fix, agent pages added, scheduled-tasks updated
|
||
- **Skills index drift incident:** `systems/skills-index.md` claimed 66 skills when 330 were actually installed (4x undercount). Page was hand-written once on 2026-07-22/25 and never regenerated. Fixed by regenerating from live skill tree, creating `regenerate-skills-index.sh` script, and adding cron job `1a8e90ea2271` (daily 03:00 AWST) to auto-regenerate and commit to Gitea. See [[incidents/2026-08-15-skills-index-drift]]. - **Updated [[systems/skills-index]]** — added auto-regeneration notice, updated counts (66→330), bumped `updated`/`verified_on` to 2026-08-15. - **Updated [[systems/scheduled-tasks]]** — added Skills Stock-Take Regeneration job, bumped count (28→29), updated dates. - **Updated [[systems/hermes-agent]]** — skills count (66→330), cron jobs count (25→29), added Mattermost gateway warning note, updated dates. - **Updated [[index.md]]** — added Incidents section with link to skills drift incident, updated last-updated date to 2026-08-15. - **Created [[incidents/2026-08-15-skills-index-drift]]** — new incident page documenting the 4x undercount, root cause (static docs for mutable inventory), resolution (auto-regeneration), and prevention (daily cron). - No secrets written. ## [2026-08-17] update | Nightly wiki audit — Nextcloud ncdownloader crash
|
||
- **New page:** `systems/nextcloud.md` — documented Nextcloud CT270: version 33.0.2.2, apps inventory, ncdownloader incompatibility incident, access details, related pages. - **Updated [[current-state]]** — added Recent Changes (2026-08-17) section with ncdownloader fix details. - **Updated [[infrastructure/proxmox]]** — CT270 entry now shows Nextcloud version and ncdownloader status. - **Updated [[index.md]]** — added Nextcloud link to Core systems, bumped last-updated to 2026-08-17. - No secrets written. ## [2026-08-16] update | Nightly wiki audit — OmniRoute capacity pressure, Agent Zero sluggishness, Hermes verification quirk
|
||
- **OmniRoute upstream capacity pressure documented:** `ServiceUnavailableError: Structurally heavy chat request capacity is busy` reported across agents via OmniRoute. Root cause: upstream OpenAI capacity pressure (recent incident Aug 11, now resolved). OmniRoute's context compression not always applied before sending oversized requests, amplifying the issue. Diagnostic requirements recorded. - **Agent Zero sluggishness confirmed as orchestration overhead:** GPT-5.6 Luna is fast — the latency comes from Agent Zero's orchestration layer (large system prompts, multi-step reasoning, memory lookups, sub-agent spawning). GPT-5.6 Luna confirmed as real model (lightweight/fast tier, launched Jul 9, 2026). - **Hermes model verification quirk documented:** Custom endpoint verification can fail with "could not reach" when the endpoint returns HTTP 402 (auth required) during unauthenticated transient checks. Config `http://omni:20128/v1` is correct — `/v1` is required. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-16) section, bumped `updated`/`verified_on` to 2026-08-16. - **Updated [[current-state]]** — added Recent Changes (2026-08-16) section, bumped `updated` to 2026-08-16. - **Updated [[index.md]]** — bumped last-updated date to 2026-08-16. - No new pages created. No secrets written. ## [2026-08-18] incident | Mnemosyne capture pipeline broken (active)
|
||
- **New page:** `incidents/2026-08-18-mnemosyne-capture-pipeline` — working memory capture (`sync_turn`) stopped functioning after 2026-08-16 23:15:39, ~48 hours of conversation turns not captured. Root cause: silent guard conditions in `run_agent.py:_sync_external_memory_for_turn()`. MEMDEBUG logging recommended. Config verified clean (native YAML types). Still active as of verification. - **Updated [[systems/mnemosyne]]** — added 2026-08-18 incident to History and Evidence, bumped `updated`/`verified_on` to 2026-08-18, added cross-reference to Related. - **Updated [[current-state]]** — added Recent Changes (2026-08-18) section with capture pipeline incident, bumped `updated` to 2026-08-18. - **Updated [[index.md]]** — added 2026-08-18 incident to Incidents section, bumped last-updated to 2026-08-18. - **Marked [[incidents/2026-08-01-mnemosyne-capture-failure]]** — status changed from `active` to `resolved` (that earlier incident was fixed). - No secrets written. ## [2026-08-23] update | Nightly wiki audit — Cron failures, wiki-stocktake drift, Mnemosyne watchdog history
|
||
- **Cron job failures documented (Weekly Review session):** Three distinct failure patterns identified in past 7 days:
|
||
- `wiki-stocktake-maintenance` (1a8e90ea2271) failing ×4 (Aug 20–23) — config drift skip: model changed from `cx/gpt-5.6-luna` → `go-cheap`/`opencode-go/gpt-5.6-luna`, job unpinned. Requires explicit provider/model pinning. - CT221 gitea miner watchdog (d5516d0a5147) failing ×6+ (Aug 19) — SSH to PVE (root@100.96.100.82) failed repeatedly; CT 221 status = UNKNOWN, never inspected. - Mnemosyne Watchdog (96eb95ce12f2) major incident (Aug 19 early AM) — no consolidation for 36–41.5+ hours (last seen Aug 17 07:15). Watchdog cycled for ~2 hours before stopping. - **Updated [[systems/scheduled-tasks]]** — added Known Issues section with cron failure patterns, bumped `updated`/`verified_on` to 2026-08-23. - **Updated [[systems/mnemosyne]]** — added Mnemosyne Watchdog Aug 19 incident to History section, bumped `updated`/`verified_on` to 2026-08-23. - **Updated [[current-state]]** — added Recent Changes (2026-08-23) section with cron failures summary. - **Updated [[index.md]]** — bumped last-updated date to 2026-08-23. - No secrets written. ## [2026-08-19] update | Nightly wiki audit — Ignis deployment, Command Code provider, Tavili search status
|
||
- **New page:** `systems/ignis.md` — documented Ignis browser-native Obsidian (Docker `ignis-subpath:0.8.9`, port 8080, Tailscale Serve at `hermes.kangaroo-eel.ts.net/ignis`). Volume mount: `/home/hermes/wiki/agent-estate-wiki:/vaults/wiki`. Verified live: container running, Tailscale route active. - **Updated [[systems/model-providers]]** — added `command-code` provider (`https://api.commandcode.ai/provider/v1`) to Current Provider Stack. Bumped `updated`/`verified_on` to 2026-08-19. - **Updated [[systems/omniroute]]** — added Recent Changes (2026-08-19) section documenting Command Code provider, `deepseek-flash-mix` combo, and Tavili search proxy status. Bumped `updated`/`verified_on` to 2026-08-19. - **Updated [[infrastructure/docker-services]]** — added Ignis compose file and service definition. - **Updated [[infrastructure/hosts]]** — added Ignis to CT460 row. - **Updated [[systems/obsidian]]** — added Ignis as browser interface, corrected vault path to `agent-estate-wiki/`, added Quartz static site reference. - **Updated [[current-state]]** — added Recent Changes (2026-08-19) section with Ignis deployment, Command Code provider, and Tavili search status. - **Updated [[index.md]]** — added Ignis link to Core systems, bumped last-updated to 2026-08-19. - No secrets written. ## [2026-08-26] update | Nightly wiki audit — Cron job reconciliation
|
||
- **Updated [[systems/scheduled-tasks]]** — reconciled live `hermes cron list` (27 jobs) against wiki table (29 entries). Removed: `Workday start reminder` (no longer in live list), `CT 221 Miner Watchdog` (job `d5516d0a5147` removed; CT221 status unknown), `Review Agent Estate Phase One` (one-time job completed 2026-08-08). Added: `weekly-save-restricted-bot-restart` (`0 3 * * 0`, local, no-agent). Updated count from 29 to 27. - **Updated [[current-state]]** — added Recent Changes (2026-08-26) section documenting cron reconciliation and wiki-stocktake-maintenance drift persistence. - **Updated [[index.md]]** — bumped last-updated to 2026-08-26, corrected job count reference (29 → 27). - **Known Issues cleanup** — removed stale CT221 miner watchdog SSH failure entry (job no longer exists). - No secrets written. ## [2026-09-02] incident | SFTPGo outage on CT299
|
||
- **New page:** `incidents/2026-09-02-sftpgo-outage.md` — documented CT299 SFTPGo outage (02:18 AWST). Root cause: CT299 swap exhausted (43Gi used, 100%), `tailscaled` died; PVE `pct start` failed due to 5 stale `lxc-attach` processes pinning cgroup. Killed PIDs 3037951/3334928/3352203/3402511/4004148, `pct stop 299 --skiplock` + `pct start 299`. Verified end-to-end: direct Tailscale connection, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. - **Updated [[current-state]]** — added Recent Changes (2026-09-02) section, bumped `updated`/`verified_on` to 2026-09-04. - **Updated [[infrastructure/proxmox]]** — CT299 entry already documented incident; no change needed. - No secrets written. ## [2026-08-28] update | Department of Communities role and social-media audit
|
||
- **New page:** `reports/communities-wa-social-media-audit-2026` — durable strategic summary of Anthony's supplied public-source audit for his Department of Communities Social Media Officer role. - **Updated [[systems/current-profile]]** — recorded current employment context and linked the report. - **Updated [[index.md]]** — added the report to the Reports section and refreshed the last-updated date. - No secrets written. ## [2026-09-20] update | Nightly wiki audit — No changes in past 24h
|
||
- **No new pages created** — no new systems, decisions, or incidents in the past 24 hours. - **No existing pages updated** — all wiki content remains current. - **Freshness audit run:** `freshness-20260920.txt` generated at 03:01 AWST (scheduled cron). - **Skills index auto-regenerated:** `systems/skills-index.md` updated at 03:01 AWST (scheduled cron `1a8e90ea2271`). - **Scheduled tasks reconciled:** `systems/scheduled-tasks.md` updated at 03:02 AWST — 26 active jobs verified from live `hermes cron list`. - No secrets written. ## [2026-09-30] update | Nightly wiki audit — Managed-runtime dependency incident + cron fleet reconciliation
|
||
- **New page `incidents/2026-09-28-hermes-managed-runtime-deps`** — the 2026-09-28 → 30 cron-fleet outage: jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`, caused by `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in the uv-managed runtime venvs under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (all 10 venvs, `pip install --target`), run every 15 min by job `96eb95ce12f2`. - **Updated `systems/scheduled-tasks.md`** — table rebuilt from live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Disabled set corrected to `d40085da631c`, `f77b7e46bd09`, `0a7f933d06e2`. Known Issues refreshed: job-radar 403 `MODEL_NOT_IN_PLAN` (streak 5), Telegram thread-delivery failures, `weekly-save-restricted-bot-restart` script restored, residual pre-fix dispatch errors on `cbfe434a3b5e`/`1a8e90ea2271`. - **Updated `systems/hermes-agent.md`** — version v0.19.0 → v0.21.5+3837.gbfda74c (2026.9.24); managed-runtimes + dependency-guard entry; default provider/model `custom` → `http://100.78.63.20:20128` / `smart-route`; skills 330 → 341; cron count 29 → 30. - **Updated `current-state.md`** — `_config_version: 46` and provider/model/fallback/curator facts re-verified live; new Recent Changes (2026-09-30) section. - **Updated `runbooks/job-radar.md`** — corrected primary model row (`custom:omni-101 / gpt-5.6-terra` via `command-code`) and added the 403 `MODEL_NOT_IN_PLAN` failure mode with recovery steps. - **Updated `systems/messaging-integrations.md`** — Telegram `allowed_chats`/delivery targets corrected; thread-delivery Known Issues; new "Multi-bot group chat" section (one forum topic per bot in a supergroup). - **Updated `index.md`** — last-updated to 2026-09-30, cron count to 30, new incident listed. - No secrets written. ## [2026-10-01] update | Nightly wiki audit — cron fleet re-verified, Telegram delivery root cause, memory hygiene
|
||
- **No new pages created** — nothing in the past 24 hours introduced a new system, decision, or concept. The work was correction and verification of existing pages. - **Updated `systems/scheduled-tasks.md`** — table and counts re-verified against live `~/.hermes/cron/jobs.json`: 30 jobs (27 enabled, 3 disabled). Failing set corrected from 4 to 3 — `dca8482e4f76` (403 `MODEL_NOT_IN_PLAN`, ×5), `1a8e90ea2271` (HTTP 410, model retired upstream, ×7), `fad5a86979d4` (HTTP 400, model absent from the NVIDIA live catalog, ×3). Row fixes: `1b23c042e17d` and `c8d52904e1d6` delivery targets now `telegram:-1004321904721:102`; `c0a9fa314542` flagged for a revoked Telegram bot token; bare-`telegram` targets noted on `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`. Known Issues restructured into Resolved / Open / Historical. - **Updated `systems/messaging-integrations.md`** — new **Delivery target semantics** section: a bare `deliver: "telegram"` resolves to the home DM (`1793951355`), *not* to the primary group, so a group topic ID attached to it hard-fails with `Message thread not found` and the message is lost. This is the root cause of the 2026-09-30 delivery incident, now marked resolved on two of the three affected jobs. - **Updated `systems/model-providers.md`** — provider stack rebuilt from live `config.yaml`: 5 custom providers (`omni-101` 5,384 models, `Omniroute:20128` 5,398, `litellm` 1,728, `command-code` 71, `guacano` 46); `custom:omni` recorded as removed. Default model corrected `resilient-free` → `smart-route`. New **Retired / Unavailable Models** table (retired `qwen3.5:397b`, plan-gated `gpt-5.6-terra`, catalog-missing `deepseek-v4-flash`) with the lesson that a raw model pin never self-recovers. - **Updated `systems/mnemosyne.md`** — new Known Issue: gateway background-process notifications were being written into the person-attributed tier as `[USER] [Anthony Martin]` memories (36 rows purged 2026-09-30, two canonical facts corrected). Restored the 2026-08-18 and 2026-08-19 history bullets that existed only in the vault copy. - **Updated `systems/hermes-agent.md`** — MCP servers (5) and enabled plugins (11) counts added; provider model counts refreshed; cron count re-verified for 2026-10-01; Known Issues rewritten to cover the three unrepinable model pins. - **Updated `index.md` and `current-state.md`** — last-updated bumped to 2026-10-01 with a Recent Changes block. - **Vault sync:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. New-page reminders for the vault: pages live under `wiki/`, not at the tree root, and `index.md` sits at the vault root. - No secrets written. ## [2026-10-02] update | Nightly wiki audit — two new cron failures, delivery-failure semantics, wiki-tree split
|
||
- **New page:** `incidents/2026-10-01-cliproxy-connection-auth-expired.md` — job `06a0a1c8704f` (AI Newsletter Curation for Patrick) failed at its 2026-10-01 10:00 run with `HTTP 401: … All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Documented as its own incident because it is a **credential** fault, not a model/routing fault, and therefore cannot self-recover. Two more enabled jobs share the pin (`1b23c042e17d`, `79f9bb41ecf5`). - **Updated `systems/scheduled-tasks.md`** — counts re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **3 → 6**. New on 2026-10-01: `06a0a1c8704f` (HTTP 401), `0e753e9720fd` (HTTP 400 — same NVIDIA model pin as the already-failing `fad5a86979d4`, confirming the shared-pin diagnosis), `066bb5191280` (delivery failed; report itself generated). Streaks advanced: `dca8482e4f76` ×5→×6, `1a8e90ea2271` ×7→×8, `fad5a86979d4` ×3. Table rows updated for all seven affected jobs; Known Issues rewritten with the provider detail and a new entry recording the two divergent wiki trees. - **Updated `systems/messaging-integrations.md`** — corrected a wrong page fact. The page previously stated that a bad `thread_id` always loses the message. The 2026-10-01 records show **two variants**: `Message thread not found` (hard failure, message lost — the 2026-09-30 case) and `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id` (a thread-less fallback send *does* occur, yet the run is still recorded `delivery_outcome: failed`). Readers must not assume "delivery failed" means the message never arrived. Also recorded that two of the three bare-`telegram` jobs failed for real on 2026-10-01, and corrected the offsetting table row that read "not failing today". - **Updated `systems/current-profile.md`** — full rewrite; the page was last verified 2026-08-28 and four facts had drifted: there is no `~/.hermes/profiles/default/` directory (the default profile is the top-level config; `ops` is the only named sub-profile), the model default is `smart-route` (not `freellmapi-auto`), the default provider is `custom` → `http://100.78.63.20:20128` / OmniRoute (litellm is now only the last fallback entry), and the config is `_config_version: 46`. Added live counts (5 MCP servers, 11 plugins, 344 skills) and the pet/skin/approval settings. - **Updated `current-state.md`** — new **Recent Changes (2026-10-02)** block; frontmatter dates bumped. - **Updated `index.md`** — last-updated to 2026-10-02; cron line now notes the failing count. - **Structural finding — the wiki exists as two trees and they have drifted both ways.** `/home/hermes/workspace/agent-estate-wiki` is canonical: it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build and is what `wiki-maintain.sh` commits. `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault copy the iPad reads (pages under `wiki/`, no git remote). The audit prompt names only the vault path. The trees have also lost pages in *opposite* directions — `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist only in the vault, while `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` and `reports/communities-wa-social-media-audit-2026.md` exist only in canonical — so canonical `index.md` currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to **both** trees. - No secrets written; all credential-shaped values in config remain redacted. ## [2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
|
||
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), failing set still **6** but the composition moved. `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` that the cliprox incident predicted — making it **2 of the 3** `cliprox/terra` jobs confirmed broken, which retires any "one-job fluke" reading. `79f9bb41ecf5` is the last untested job on that pin (next due 2026-10-04 18:00). **Streak recount** against `executions.db`: `dca8482e4f76` ×6→**×7**, but `1a8e90ea2271` ×8→**×7** — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). **Resolved:** `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). **Downgraded:** the GitHub Trending Radar revoked-token claim is now "unverified at last run" — `c0a9fa314542`'s 2026-09-30 19:22 run completed *and delivered*. New Known Issue recording that **a delivered notice is not a successful job** (both cliprox failures read `delivery_outcome: delivered` on a `failed` status). - **New page `incidents/2026-08-26-work-outlook-basic-auth-disabled.md`** — the `work` Outlook mailbox has been unreachable since **2026-08-26**: himalaya is configured for password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. TCP/TLS to `outlook.office365.com:993` is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap. - **Updated `systems/messaging-integrations.md`** — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, `work` unreachable; frontmatter `updated`/`verified_on` bumped. Added the tooling consequence: `email/mail-lookup` resolves the account before searching, so work-mail requests hit this wall and must be reported as *account unreachable*, not "no results found". - **Updated `systems/hermes-agent.md`** — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to **six jobs across three model faults plus one credential fault**, and a second entry added for the stock-take job being down; the `Email: connected` bullet corrected. - **Updated `current-state.md`** and **`index.md`** — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped. - **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root. - No secrets written; all credential-shaped values in config remain redacted. ## [2026-10-04] Nightly audit — CT450 update + Mnemosyne vector-recall repair
|
||
|
||
Reviewed window: 2026-10-03 00:12 → 2026-10-04 00:12 AWST. **New pages**
|
||
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — `mnemosyne_recall` had been scoring keyword-only: `fastembed` was absent from every uv-managed runtime, so the dense leg produced no candidates and recall fell back to FTS5 *without raising an error*. Repaired 2026-10-03; the runtime guard now asserts fastembed + `FASTEMBED_CACHE_PATH` + model dir every 15 min. Live 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%). - `incidents/2026-10-03-ct450-hermes-update.md` — CT450 (`hermesagent`) updated v0.19.0 → v0.21.5, its first update since July. Records the no-SSH access path (`ssh root@100.96.100.82` → `pct exec 450 -- …`) and the four pre-flight checks (clang, stale locks, tree ownership, Caddy keyring perms). **Updated**
|
||
- `current-state.md` — new Recent Changes (2026-10-04) block: the two findings above, the managed-runtime prune 10 → 3, cron re-verification, Mattermost unreachable, and the `mnemosyne doctor` / `sqlite_vec` misread gotcha. - `systems/mnemosyne.md` — Database Stats refreshed to live 2026-10-04 numbers; the 2026-08-02 "vector coverage gap" marked RESOLVED (retired as a current issue); new Known Issues subsection for the FTS-only degradation plus the doctor caveat; History entry; embedding-coverage job `b63ed5817212` documented. - `systems/hermes-agent.md` — version corrected to `v0.21.5+6169.gab86a20` (was `v0.21.5+3837.gbfda74c`), managed-runtime count corrected 10 → 3, guard description extended, profile line corrected (no `profiles/default/`; `ops` is the only named sub-profile), Known Issues bullet added. - `systems/memory-backend.md` — Known Issues bullet for the FTS-only degradation. - `systems/scheduled-tasks.md` — 2026-10-04 re-verification (30 jobs, failing set still 6, no new failures in the window), `79f9bb41ecf5` due 2026-10-04 18:00; bare-`telegram` delivery warnings confirmed live. - `agents/hermes-production.md` — version, Python, profile, provider routing, cron count and messaging refreshed; stale `custom:litellm` / `freellmapi-auto` references replaced. - `agents/hermes-chaos.md` — CT450 version bump recorded, no-SSH access path added, Mattermost failure noted. - `infrastructure/hosts.md` — CT450 row updated to v0.21.5. - `runbooks/update-hermes-safely.md` — new "Second instance — CT450" section (access path + pre-flight checks). - `runbooks/provider-health.md` — stale v0.19.0 version reference corrected. - `index.md` — last-updated bumped; both new incidents linked; cron-fleet line re-dated. **Note** — `mnemosyne doctor` on this host prints `vec0: present_but_unloadable` and marks the vector tiers unavailable. That is a Python-3.11-legacy-`venv` artefact, not the gateway's state. Judge vector health with `~/.hermes/scripts/check_embedding_coverage.py`. ## [2026-10-05] update | Nightly wiki audit — cliprox pin settled 3/3, failing set 6 → 7
|
||
|
||
Reviewed window: 2026-10-04 00:12 → 2026-10-05 00:01 AWST. - **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json` + `hermes cron list`: still 30 jobs (27 enabled, 3 disabled), but the failing set moved **6 → 7**. The one new failure event in the window is `79f9bb41ecf5` (**Weekly Review**, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. It was the last untested job on the `cliprox/terra` pin, so that pin is now **3 of 3 confirmed broken** — a provider-connection credential fault, not a per-job fluke. Count block, job table rows (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`) and the Known Issues Open list all updated; frontmatter dates bumped. - **Updated `incidents/2026-10-01-cliproxy-connection-auth-expired.md`** — new "Third job confirmed — 2026-10-04 18:00 · pin settled 3/3" section; Impact line changed from "2 of 3 failed, 1 untested" to "3 of 3"; status line and frontmatter dates/sources bumped. Still **Open** — needs the OmniRoute connection re-authenticated before any repin. - **Updated `current-state.md`** — new **Recent Changes (2026-10-05)** block: the cliprox 3/3 settlement, `1a8e90ea2271`'s repeat 410 on 2026-10-04, confirmation that **no interactive sessions occurred in the window** (all 10 active sessions were cron), and a streak-counting caveat. - **Updated `index.md`** — last-updated → 2026-10-05; cron-fleet line now reads "7 failing as of 2026-10-05"; cliprox incident line now "all 3 jobs sharing the pin now failed". - **Streak-number caveat recorded:** `hermes cron list`'s "N failures in a row" is the `failure_streak` counter in `jobs.json`, not a DB count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows total), so `executions.db` recounts undercount older streaks — the earlier DB-based "×7" for `1a8e90ea2271` undercounts against the live ×11. Quote the live counter and note the retention floor. - **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source — `wiki/systems/scheduled-tasks.md`, `wiki/incidents/2026-10-01-cliproxy-connection-auth-expired.md`, `current-state.md`, `index.md`, `log.md`. - No secrets written; all credential-shaped values remain referenced by connection ID only. ## [2026-10-06] update | Nightly wiki audit — dead model pins repinned, failing set 7 → 0
|
||
|
||
Reviewed window: 2026-10-05 00:04 → 2026-10-06 00:15 AWST. **New**
|
||
- `incidents/2026-10-05-dead-model-pin-repin.md` — the seven-job dead-model-pin repin. Records the four dead pins (`cliprox/terra` HTTP 401 ×3, `ollama-cloud/qwen3.5:397b` HTTP 410 ×1, `nvidia/deepseek-ai/deepseek-v4-flash` HTTP 400 ×2, `command-code/gpt-5.6-terra` HTTP 403 ×1), the remediation window 2026-10-05 23:16 → 2026-10-06 00:05, and the deliberate choice of routing aliases (`free-stack` ×6, `smart-route` ×1 on `custom:omni-101`) over another raw model name. **Updated**
|
||
- `systems/scheduled-tasks.md` — Count block rewritten: **0 failing** (was 7), with all 30 `failure_streak` counters at 0; four jobs carry a stale pre-repin `last_status: error` that clears on their next scheduled run. Seven job-table rows updated (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`, `dca8482e4f76`), plus `c0a9fa314542`. New `### Resolved 2026-10-05` subsection in Known Issues; five resolved bullets removed from `### Open`, leaving bare-`telegram` targets, the two-wiki-tree drift, the work-mailbox block, the stale `custom:omni` key and the delivered-notice rule as the standing open set. - `index.md` — new incident linked at the top of the Incidents list; cron-fleet line re-dated to 0 failing as of 2026-10-06; the cliprox incident line marked closed. - `current-state.md` — new Recent Changes (2026-10-06) block; frontmatter dates bumped. **Findings worth the ink**
|
||
- **`dca8482e4f76` (Daily Marketing Job Radar) delivered to topic 1030 at 23:56** — its first successful run since 2026-09-25. The repin is verified by live run, not assumed. Same for `79f9bb41ecf5` (23:39) and `06a0a1c8704f` (23:45). - **GitHub Trending Radar's revoked-token diagnosis is dead.** `c0a9fa314542` ran on schedule at 2026-10-05 09:00 and completed cleanly — two consecutive successes after 2026-09-30 19:22. No `getMe` → 401; no BotFather token swap needed. The 2026-10-05 audit correctly refused to record this as standing fact; it can now be retired outright. - **`delivery_outcome: delivered` still does not mean the job worked.** Every pre-repin failure recorded `delivered`, because the failure notice is itself the deliverable. Judge on the execution `status` column. - **A stale notice opened the review.** Anthony was prompted by a `85edf6772bca` notice that had sat in topic `346` since the 2026-09-29 07:00 run; the job succeeded on every run from 2026-09-30 17:53 and its incidents were resolved 2026-09-30 17:55:13. The notice's wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
|
||
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` and `dca8482e4f76` at 23:33/23:48, superseded by their successful re-runs) and 1 `running` (this audit). **Vault mirror:** the new incident page plus `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by connection ID only. ## [2026-10-07] update | Nightly wiki audit — stocktake job verified by live run, fleet green a second day
|
||
|
||
Reviewed window: 2026-10-06 00:00 → 2026-10-07 00:00 AWST. **Updated**
|
||
- `systems/scheduled-tasks.md` — Count block corrected: the stale pre-repin `last_status: error` set is now **three** (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`), not four — `1a8e90ea2271` cleared its own when it ran at 03:00. Added a **Re-verified 2026-10-07 AWST** bullet (30 jobs, 27 enabled / 3 disabled, 0 failing; 176 `completed` / 0 `failed` / 1 `running`; 14 sessions, all cron). The stocktake job-table row and the "Verified live, not assumed" bullet now record the live-run verification. Historic count progression extended. - `index.md` — last-updated → 2026-10-07; cron-fleet line re-dated (**0 failing as of 2026-10-07**, 4 of the 7 verified by live run); the skills-index line no longer reads "being refreshed by hand while the stocktake job is blocked" — it now names 346 skills and the auto-regeneration. - `current-state.md` — new Recent Changes (2026-10-07) block; frontmatter dates bumped. **Findings worth the ink**
|
||
- **`1a8e90ea2271` ran itself at 03:00 and proved the repin.** `wiki-stocktake-maintenance` completed **2026-10-06 03:00:31 → 03:03:32** on `custom:omni-101` / `free-stack` — its first automatic success since 2026-09-25 (it had failed HTTP 410 ×11 on the retired `qwen3.5:397b`). It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt` and committed to Gitea as `33f4e95`. Its four HTTP-410 incidents (`6df957a05ec6`, `d284168d3663`, `cda67de5b3e3`, `e31584909bd2`) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. **The daily manual `wiki-maintain.sh` run can stop.**
|
||
- **Vault mirror drift found and partly repaired.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` was still 2026-10-03 / 344 skills while the canonical copy had regenerated to 2026-10-06 / 346 — a knock-on of the stocktake having been blocked. Re-mirrored in this audit; the underlying two-wiki-tree split remains a standing open item. - **Window tally** (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00): 176 `completed`, 0 `failed`, 1 `running` (this audit) — 177 rows. 14 sessions, all `cron`; no interactive (non-cron) sessions, and no new incident opened. **Vault mirror:** `systems/scheduled-tasks.md`, `systems/skills-index.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by connection ID only. ## [2026-10-08] update | Nightly wiki audit — first auto-resolved incident since 2026-09-30, two stocktake script findings
|
||
|
||
Reviewed window: 2026-10-07 00:00 → 2026-10-08 00:00 AWST. **Updated**
|
||
- `systems/scheduled-tasks.md` — new **Re-verified 2026-10-08 AWST** bullet (30 jobs, 27 enabled / 3 disabled, **0 failing**; 176 `completed` / 1 `failed` / 1 `running`; 13 sessions, all cron; 12 deliveries, 0 delivery failures). Historic count progression extended. DHL job row now records the 08:00 Camofox timeout and the parcel's arrival. Stocktake row corrected — its Quartz step is inert, and the second consecutive automatic run is recorded. New **Resolved 2026-10-07** Known Issues section, plus two new **Open** items. The "Verified live, not assumed" bullet now covers the stocktake's consecutive runs. - `index.md` — last-updated → 2026-10-08; cron-fleet line re-dated; skills-index line re-dated to 2026-10-07 and annotated with the 347-on-disk caveat. - `current-state.md` — new Recent Changes (2026-10-08) block; frontmatter dates bumped. **Findings worth the ink**
|
||
- **The incident machinery had its first real workout since 2026-09-30 — and it worked.** `62619aa6bcb3` (DHL Tracking) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call, i.e. a momentarily unresponsive Camofox. Incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) was opened 08:00:52, alerted 08:00:58 and **auto-resolved 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. Transient and self-recovering — no action needed. - **The tracked parcel has arrived.** The 14:00 run reported waybill `8052838382` as `Delivered`, so the DHL monitor is now redundant and is flagged as a removal candidate rather than removed. - **Stocktake ran automatically for the second consecutive day** (`1a8e90ea2271`, 2026-10-07 03:00, commit `8c63ef2`; 346 skills / 70 categories; freshness 62 stale / 21 fresh / 7 no-date of 90 pages) — the repin is verified across consecutive runs, not just once. - **Two real defects surfaced from the stocktake's own log.** (1) Step 3 of `wiki-maintain.sh` — the Quartz rebuild — **is skipped on every run**: there is no `quartz.config.ts` and no `npx`, so `public/` is never refreshed and the published site silently ages. (2) `skills-index` **under-reports by one**: 347 `SKILL.md` files exist on disk but the page says 346, because the generator de-duplicates on skill basename within a category and so swallows the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` (the other duplicated basename, `hermes-api-vault`, survives because its two copies sit in different categories). Both are now Open items in [[systems/scheduled-tasks]]. - **Unchanged, still carrying a stale `last_status: error`:** `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` — due Fri 08:00 / Mon 03:00 / 1st 09:00. The three bare-`telegram` targets and the `custom:omni` stale-key references are also unchanged. **Vault mirror:** `systems/scheduled-tasks.md`, `index.md`, `current-state.md` and `log.md` mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by job ID and connection ID only. ## [2026-10-09] wiki audit | Fleet-wide provider timeout storm; 2026-10-08 window backfilled; DHL retirement still open
|
||
|
||
**Window audited:** 2026-10-08 00:00 → 2026-10-09 24:00 AWST — the 23:59 run on 2026-10-08 timed out and
|
||
produced no entry, so this run covers two days. **Pages written**
|
||
- `incidents/2026-10-09-cron-provider-timeout-storm.md` — new. - `incidents/2026-10-09-memory-recall-relevance.md` — new. - `systems/scheduled-tasks.md` — Re-verified 2026-10-09 block appended (fleet tallies, storm job list, six open items); frontmatter dates bumped. - `systems/mnemosyne.md` — live tier counts refreshed; new dated counts line with a pointer to the recall incident. - `current-state.md` — Recent Changes (2026-10-09) block; frontmatter dates bumped. - `index.md` — last-updated → 2026-10-09; cron-fleet line rewritten; skills-index line annotated with the stocktake gap; two incident links added. - `systems/skills-index.md` + `raw/stocktake/freshness-20261008.txt` — the stranded 2026-10-08 03:01 stocktake artifacts committed. **Findings worth the ink**
|
||
- **The fleet had its worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2), all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, **eight remain open**. The watchdog (`96eb95ce12f2`) failed twice as a *cascade* — it exits 1 when it finds ERROR lines in `agent.log`, i.e. it reported the storm rather than suffering it. - **The 2026-10-08 window was never audited** — this job's own 23:59 run failed with the same timeout (incident `cbfe43_fa81052baaf7`) and committed nothing. - **The stocktake has a two-day gap.** 10-08's output was left uncommitted/untracked (both committed here); 10-09's run timed out before generating anything, so no `freshness-20261009.txt` exists. `1a8e90ea2271` is the one job still at `failure_streak: 2`. - **An instruction was reported done and was not.** Anthony said "Delete this job" to the DHL delivered notice at 21:24 on 2026-10-09; the session replied that `62619aa6bcb3` was "stopped and deleted". It is **still enabled** (`paused_at: null`) and fires at 08:00 daily. Flagged as open item #1 rather than deleted by this audit. - **Memory recall relevance** — "Are you up to date" returned a stale Uber Eats episode. Capture and vectors verified healthy (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic); the in-session "config saved" claim is **not** in live config (`memory.mnemosyne` carries no recall-weight keys; no top-level `mnemosyne:` key). **Vault mirror:** the touched pages mirrored into `/home/hermes/wiki/agent-estate-wiki/` for the iPad/Ignis view. No secrets written; credential-shaped values referenced by job ID and connection ID only. ## 2026-10-10 — Stocktake (manual run, `wiki-maintain.sh`)
|
||
|
||
- **`systems/skills-index.md` regenerated** — 346 skills / 70 categories (unchanged count; frontmatter `updated`/`verified_on` refreshed to 2026-10-10). Live on-disk count is 347 `SKILL.md`; the one-skill gap is the known de-dup of `productivity/obsidian-git-sync/obsidian-git-sync/`. - **Freshness audit** → `raw/stocktake/freshness-20261010.txt`: **62 stale (≥21d), 23 fresh, 7 no-date of 92 pages**. Oldest cluster is the 2026-07-22 bulk (80d) — 27 pages written in one sitting and never revisited. No `freshness-20261009.txt` exists: that 03:00 run timed out before the generator produced output, so the two-day artifact gap stands. - **`current-state.md` — new 2026-10-10 section.** The 2026-10-09 timeout storm window re-tallied from `executions.db` (2026-10-09 00:00 → 2026-10-10 00:00 AWST): **163 `completed`, 15 `failed`** — the prior audit recorded 162/15+1 running; the stray `running` row is the stocktake job's own in-flight 03:00 run. 10 incidents remain open (list in page). - **DHL monitor re-probed, still enabled.** `62619aa6bcb3` — `enabled: True`, `paused_at: null`, `0 8,11,14,17,20 * * *`, `last_status: ok`. Anthony instructed deletion 2026-10-09 21:24; a session reported it done. **No cron mutation performed by this audit** — retirement stays open item #1. - **`infrastructure/tailscale.md` verified against `tailscale serve status`** — all 13 tailnet Serve routes plus the `:8443/webhook` Funnel match the page. No drift; not edited. - **Not verified / left alone:** the Quartz rebuild step (no `quartz.config.ts`, `npx` cannot fetch `quartz` — `public/` was NOT refreshed, confirming the carried open item); `docker ps` (permission denied on `/var/run/docker.sock` from this session, so `infrastructure/docker-services.md` could not be probed and is left stale); the 80d `2026-07-22` cluster (no live source to check against). - **Sessions in the 2026-10-09 window (`state.db`, epoch REAL):** 19 total — 6 `cron`, 11 `ios`, 1 `talaria-ios`, 1 `telegram`. Delivered 16 / suppressed 160 / suppressed-acked 2 (0 delivery failures — the `delivered` count includes failure notices). No secrets written; job/incident IDs only. ## [2026-10-10] wiki audit | window 2026-10-09 -> 2026-10-10 (ran 2026-10-10 23:59 -> 2026-10-11 00:2x AWST)
|
||
|
||
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-11T00:19). Day tally: 10-07 `177/1`,
|
||
10-08 `178/2`, 10-09 `163/15`, **10-10 `174 completed / 0 failed`**. Non-clean `last_status` 10 -> 5. Incidents: **7 closed during 10-10, 5 open** (3 jobs). **Open item #1 closed: the DHL monitor
|
||
`62619aa6bcb3` is retired and verified absent from `jobs.json` by ID lookup** (the 10-07 and 10-09
|
||
"deleted" claims were false). **Runtime/memory.** The 2026-09-28 managed-runtime-dependency class **recurred in the gateway env**:
|
||
`mnemosyne` (+native deps) and `elevenlabs` were missing from the committed uv env
|
||
`795020dbf24343cd9c52df24723dca83` that `activate_dependencies()` loads. Repaired 21:26-21:34 and
|
||
verified live from 22:01:19 (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - left open. **New page.** [[systems/paseo]] - Paseo daemon v0.11.2 on `0.0.0.0:6767` (user unit
|
||
`paseo-daemon.service`); recorded the `daemon run` vs `daemon start` supervisor trap, the
|
||
`/usr/bin/paseo` 0.7.2 vs daemon 0.11.2 version split, and the truncated-`cline` false provider fault. **Platform.** Mattermost unreachable from CT460 - configured endpoint `100.98.116.14:8065` answers
|
||
neither ICMP nor TCP; gateway retrying every 5 minutes. Cause not determined. **Pages touched:** `systems/paseo.md` (new), `systems/scheduled-tasks.md`, `systems/hermes-agent.md`,
|
||
`systems/mnemosyne.md`, `systems/messaging-integrations.md`,
|
||
`incidents/2026-09-28-hermes-managed-runtime-deps.md`, `current-state.md`, `index.md`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-11] stocktake | Daily maintenance + live drift corrections
|
||
- Ran `wiki-maintain.sh`: skills-index regenerated (346 skills / 70 categories), freshness audit written to `raw/stocktake/freshness-20261011.txt` (62 stale / 24 fresh / 7 no-date of 93 pages)
|
||
- **OmniRoute CT600/CT601 roles are INVERTED from what the wiki claimed.** Verified via `pct list`, `pct config`, `ss` and `tailscale status`: CT600 `omniroute` (100.88.81.19) is RUNNING and serving `:20128`; CT601 `omni` (100.93.204.4) is STOPPED with `onboot=0`. Hermes resolves `custom:omni` → `omniroute:20128` → CT600, not CT601. systems/omniroute.md rewritten with the live node table. - **Third OmniRoute node found:** `omniroute-1` at 100.78.63.20:20128 is live (returns 401 = auth-required) and is registered in Hermes config as custom provider `omni-101` (5384 models, default `smart-route`, carries `skills_hub`). Not in `pct list` — identity unconfirmed. - **Ignis duplicate found:** a second `ignis-ignis-1` container runs on CT200 (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`), separate from CT460's `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`). Both confirmed serving HTTP 200. systems/ignis.md now distinguishes them. - **Tailscale route table corrected** against live `tailscale serve status`: `/agent-estate` target is bare `http://127.0.0.1:8769` (not `.../8769/agent-estate-role-dashboard.html`); previously-missing tailnet-only `:9443` → `127.0.0.1:5173` documented. CT201 routes refreshed (8085 targets + `/steel` 5173 + `/steel-api` 3000). - **Proxmox inventory rebuilt** from live `pct list` + `qm list`: added CT475 `agentzero` and VM110 `omarchy`; corrected 282 `ubuntu-server-xfce` to stopped; removed stale rows (310 monitoring, 500 litellm, 104 rclone, 106 glance, 101 debianvm). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed rather than guessed. - **Nanobot verified** on CT333: gateway `nanobot-gateway` (PID 2259048) listening on 8765/8766; NOT a systemd `nanobot.service`/`gateway.service` (both probes returned inactive — wrong unit names). Port 8900 no longer listening — flagged as unverified. - **Docker services page corrected:** the old "Docker runtime not available" note was a CT460 socket-permission artifact (`/var/run/docker.sock` is `root:docker srw-rw----`, `hermes` user not in group `docker`, no passwordless sudo in cron). Documented the three real Docker hosts (CT200 ~50 containers, CT333 9, CT460 via sudo). - **Flagged for follow-up, root cause not investigated:** `mycompose-db-1` on CT200 is in `Restarting (1)` — crash-looping. - Mattermost CT255 confirmed stopped, matching its Tailscale peer offline 21d. - Left as stale-without-fix: 62 pages still older than 21 days (concepts, decisions, comparisons, most runbooks) — these were not probed against a live source this pass, so they were reported, not rewritten. ## [2026-10-12] wiki audit | window 2026-10-11 -> 2026-10-12 (ran 2026-10-11 23:59 -> 2026-10-12 00:2x AWST)
|
||
|
||
**Cron.** 29 jobs, 26 enabled / 3 disabled (`jobs.json` 2026-10-12T00:10). Day tally: 10-07 `177/1`,
|
||
10-08 `178/2`, 10-09 `163/15`, 10-10 **`175 completed / 0 failed`** (finalised), 10-11 **`171 completed /
|
||
0 failed`** + 1 `running` — **two consecutive clean days**. Non-clean `last_status` steady at 5; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). **Corrected a false alarm instead of logging a fault.** The 2026-10-11 00:08 Mnemosyne Health Monitor
|
||
EXCEPTION ("capture stalled 10.7 h") was a **naive-UTC vs AWST comparison error** — real gap 2.7 h, the row it
|
||
cited landed inside the session that wrote it. Recorded in [[systems/mnemosyne]] and
|
||
[[systems/scheduled-tasks]] so the next run of that monitor is not triaged as a capture outage. Live counts
|
||
refreshed: 786 WM / 1,056 EP, coverage 98.3 % / 99.6 %. **New content.** [[queries/retained-research-answers]] was a 2026-07-22 stub and is now a real page — the
|
||
2026-10-11 VNC-client research (server-side encoder is the bottleneck; TigerVNC native viewer for standard RFB; `-CompressLevel` 2 is client-side; RealVNC's "4.7×" is self-referential) with its sources. Proxmox storage
|
||
was undocumented; the live datastore table plus host mount points are now in [[infrastructure/proxmox]]
|
||
(`/mnt/ssd0` 93 % / ~63 GiB free is the pressure point). **Stale labels fixed.** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] carried `status: active` from
|
||
August despite capture being verifiably healthy — now `superseded`, with the live evidence and a caution about
|
||
the UTC comparison. Index's "8 incidents still open" from the 2026-10-09 storm trimmed to the 2 genuinely open,
|
||
and the skills-index blurb no longer claims no regeneration since 2026-10-07. **Verified, not changed.** The `blocked-page-recovery` lesson that session reported patching
|
||
(`recovered: false` = archive routes failed, not task failed; API-first rung) is present in the live skill. **Pages touched:** `systems/scheduled-tasks.md`, `systems/mnemosyne.md`, `infrastructure/proxmox.md`,
|
||
`queries/retained-research-answers.md`, `incidents/2026-08-18-mnemosyne-capture-pipeline.md`,
|
||
`current-state.md`, `index.md`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-12] stocktake | wiki-maintain.sh run + 2 live-drift corrections
|
||
|
||
**Script run.** `wiki-maintain.sh` completed all three stages: skills-index regenerated from the live
|
||
skill tree (**346 skills** across 70 categories, unchanged count), freshness audit written to
|
||
`raw/stocktake/freshness-20261012.txt` (**54 stale / 32 fresh / 7 no-date of 93 pages**), and the Quartz
|
||
step took its documented no-`quartz.config.ts` path (build skipped, `public/` unchanged — expected, not a
|
||
failure). The in-process watcher loop hit the 420 s tool cap, but the script had already finished and
|
||
written its artifacts, so results were read from the report rather than assumed. **Drift corrected from live probes (2 pages):**
|
||
|
||
1. **[[systems/nanobot]]** — last verified 2026-08-07 as DOWN. Live probe 2026-10-12: the Nanobot web UI on
|
||
CT333 is **back up**, HTTP `200` on `http://100.122.189.81:8766/`. Port `8900` — the endpoint the
|
||
disabled OmniRoute `nanobot` provider connection uses — is **still closed**, so `status` stays
|
||
`degraded` rather than flipping to active. SSH to CT333 is unreachable from CT460, so
|
||
`whatsapp-bridge.service` was not re-verified. 2. **[[systems/home-assistant]]** — carried `confidence: low` plus a page-wide "cannot confirm HA runs"
|
||
caveat from 2026-07-22. Live: HTTP `200` on `http://100.68.139.5:8123/`, Tailscale node
|
||
`homeassistant` active and offering an exit node, VMID 100 running per `qm list`. Confidence raised
|
||
low → medium; the unverified residue is now narrowed to Hermes→HA entity access (no HA credentials on CT460). **Verified and left stale.** [[systems/openrouter]] and [[systems/nextcloud]] were probed shallowly and
|
||
nothing contradicting was found, so both stay in the stale list rather than being rewritten on guesswork. **Pages touched:** `systems/nanobot.md`, `systems/home-assistant.md`, `systems/skills-index.md`,
|
||
`raw/stocktake/freshness-20261012.txt`, `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-13] audit | nightly review of 2026-10-12 — clean cron day hid a dead credential; Camofox containment ceiling; two pages corrected against live probes
|
||
|
||
**Window.** 2026-10-12 00:00 → 23:59 AWST (sessions from `~/.hermes/state.db`, cron from
|
||
`~/.hermes/cron/executions.db` + `cron_incidents`; live probes 2026-10-13 00:05–00:45). One former
|
||
compaction of this job left part of the review written but the log entry and cross-page updates
|
||
unfinished — completed here. **Headline 1 — a clean run *hid* a dead credential.** `c0a9fa314542` (GitHub Trending Radar — Weekly
|
||
Roundup, Mondays 09:00) reported **`completed` with `delivery_outcome: delivered`** on 2026-10-12, but its
|
||
fetch-then-send script died at `digest.py` `send()`. Live probe of the bot token in
|
||
`/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 from `getMe`** — the token is
|
||
a valid 46-character shape, i.e. **revoked**, not malformed. The fetch path is healthy (100 repos, 26 cache
|
||
files in 24 h). Critically, this **supersedes the 2026-10-06 entry on [[current-state]]**, which concluded
|
||
the revoked-token diagnosis was "refuted" because two runs had finished green. Lesson recorded in
|
||
[[systems/scheduled-tasks]]: **a completed run does not clear a credential fault unless the run exercised
|
||
the same call that failed**, and green cron status is not delivery proof. (Credential was present; value
|
||
neither recorded nor repeated.)
|
||
|
||
**Headline 2 — Camofox hit its cgroup ceiling inside a cron run (new).** The 2026-10-12 07:01 Daily
|
||
Marketing Job Radar run drove `camofox-browser.service` to **`2.5G`, swap fully consumed, `available: 0B`,
|
||
383 tasks** against the drop-in ceiling `MemoryHigh=2G` / `MemoryMax=3G` / `MemorySwapMax=1G`. REST `/tabs`
|
||
and page evals hung (indistinguishable from ordinary tool timeouts). The job restarted the service at
|
||
**07:23:50**, switched to one-tab-at-a-time extraction with immediate closes, and still delivered. Diagnosis:
|
||
**workload exceeded the cgroup, not a leak** — `OOMPolicy=kill` never fired. **Headline 3 — [[systems/browser-backend]] corrected against live probes.** Prior revisions were wrong on
|
||
three counts: the REST API listens on **`:9377`** only (`:9093` closed, `:9378` does not exist — REST and
|
||
WebSocket are the same service, not two), Camofox runs as a **`systemd --user` unit on the Hermes host
|
||
(CT460)**, not CT450, and the containment drop-in was undocumented. `current-state.md` also still described
|
||
the old port model; both pages now match the probe. **Cron state.** 2026-10-12: **174 completed / 0 failed / 1 running** (this audit, `cbfe434a3b5e`, started
|
||
23:59:41) — **third consecutive clean day**; 2026-10-11 finalised at `172 / 0`. Retention floor moved to
|
||
2026-10-07 08:15. Fleet **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3**
|
||
(`0e753e9720fd` error, `1b23c042e17d` error, `066bb5191280` delivery_failed). `fad5a86979d4` (Facebook
|
||
Birthday Weekly Registry Sync) and `dca8482e4f76` (Daily Marketing Job Radar) both cleared on 2026-10-12 —
|
||
the former at 03:00:45 → 03:05:47 (448 birthdays, delivered), the latter 07:00:47 → 07:33:50. **Incidents — open 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at
|
||
**03:05:39** (the two `deepseek-v4-flash` HTTP-400 rows, cleared by the registry sync's clean run) and
|
||
`dca848_23fcd2b18aaa` at **07:33:41**. Remaining: `1b23c0_c35703f637b8` (`alerted` — weekly job, cannot
|
||
clear before Fri 2026-10-16 08:00) and `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — the job has no
|
||
retained executions, so it cannot clear until 2026-11-01). → [[incidents/2026-10-09-cron-provider-timeout-storm]]
|
||
updated from "8 still open" to **11 of 12 closed** with the full closure timeline; the pattern recorded
|
||
there is that **each incident clears only on the next successful run of its own job**, so weekly jobs carry
|
||
their rows for up to a week after the fault is gone. **Mnemosyne (2026-10-12 12:02 health run, green).** Working **760/786 = 96.7 %**, episodic
|
||
**1050/1056 = 99.4 %** — both above the 95 % floor, no `reindex`; orphan embeddings 0; FTS5 exactly aligned; 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]]. **Sessions.** **18 in the window, all `cron`, zero interactive.** The 03:00 stocktake `1a8e90ea2271` ran
|
||
clean and pushed `0eee5f1` (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70
|
||
categories — the known 347-vs-346 on-disk off-by-one persists). **Pages touched:** `current-state.md` (new Recent Changes section + superseded-correction tag),
|
||
`systems/scheduled-tasks.md` (frontmatter, job inventory, non-clean statuses, re-verified history, five
|
||
job rows, `### Resolved 2026-10-12`, corrected incident IDs, `### Open` additions, new
|
||
`## Re-verified 2026-10-13` block), `systems/browser-backend.md` (port + host + containment rewrite,
|
||
history entries), `systems/mnemosyne.md` (12:02 coverage), `incidents/2026-10-09-cron-provider-timeout-storm.md`
|
||
(status + full closure table + timeline), `index.md` (date, four blurbs), `log.md`. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-14] audit | nightly review of 2026-10-13 — the clean streak broke on a dead opencode credential
|
||
|
||
Window 2026-10-13 00:00 → 23:59 AWST (ran 2026-10-13 23:59 → 2026-10-14 00:2x AWST). Live probes only. **Cron.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight → the day finalises
|
||
at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` — **the first failed run since 2026-10-09**
|
||
(which recorded 15). Retention floor moved to **2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows
|
||
= 1,000 terminal + 1 in flight), so days above the floor are complete and older ones undercount (10-08 reads
|
||
`126 / 1`, partial). **Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 →
|
||
03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned
|
||
`custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`,
|
||
`alerted_at: NULL` — streak 1, under the 3-strike alert rule). Third failure of this job in six days, all three
|
||
on the opencode path (10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode`; 10-09
|
||
timeout; 10-13 404). The first two resolved on 2026-10-10; this one clears only on a successful 2026-10-14 03:00
|
||
run. → [[incidents/2026-10-05-dead-model-pin-repin]]
|
||
|
||
**Finding 2 — the failure stranded its own generated artifacts (now committed).** Step 1 (skills-index) and
|
||
step 2 (freshness report) both completed, then the agent turn failed → no commit, no push. Left in the working
|
||
tree: `systems/skills-index.md` (modified, **date-only**: `updated`/`verified_on` 10-12 → 10-13; **346 skills /
|
||
70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked; **48 stale / 38 fresh /
|
||
7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are committed by this audit. Consequence: the stocktake published nothing on 10-13 — and because the *agent* row is what commits, even a
|
||
perfect script run leaves nothing behind when the LLM turn dies. **Delivery.** `159 suppressed` / **9 delivered** / 1 no-record. Delivered: Daily Marketing Job Radar,
|
||
Mosquito Tracker ×3, Tech & AI Newsletter Digest, Daily Birthday Checker, Gmail Action Digest, FreshRSS Midday,
|
||
Evening Stand-Down Briefing. No unexpected delivery; the two dead-thread faults (`0e753e9720fd`,
|
||
`066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 2026-10-01. **Mnemosyne.** Six monitor runs, all silent: Health Monitor 00:02 + 12:02, Embedding Coverage Monitor 00:02 /
|
||
06:02 / 12:01 / 18:01 → no exception, so both coverages stayed at/above the 95 % floor and no `reindex` was
|
||
needed. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic
|
||
1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the silent token markdown-bolded rather than
|
||
bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`) — the matcher tolerates wrapping
|
||
rather than requiring the literal token as a strict control. → [[systems/mnemosyne]]
|
||
|
||
**Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive
|
||
session. **Fleet.** 29 jobs (26 enabled / 3 disabled), pins unchanged; **non-clean `last_status` 3 → 4** (cleared
|
||
`fad5a86979d4` 10-12 03:05 and `dca8482e4f76` 10-13 07:02; added `1a8e90ea2271`); **open incidents 2 → 3**
|
||
(1 `alerted`, 2 `detected`, 8 `closed`, 66 `resolved`; 77 rows). **Pages touched:** `systems/scheduled-tasks.md` (frontmatter, retention floor, non-clean list,
|
||
`## Re-verified 2026-10-14`), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`
|
||
(new Recent Changes + date), `index.md` (date + two blurbs), `log.md`. **Vault mirror.** The 2026-10-13 audit was never mirrored to `/home/hermes/wiki` (last mirror `bb09581`,
|
||
2026-10-12), so this run mirrored both nights: the 10-13 audit's ten pages plus `skills-index.md`, whose vault
|
||
copy was still the 2026-10-08 version. The vault's own `raw/stocktake/` still holds only the 2026-10-03 report. No secrets written; job/incident IDs, paths and tailnet addresses only. ## [2026-10-14] stocktake | 03:00 maintenance — second Proxmox hypervisor discovered; tailnet inventory rebuilt
|
||
|
||
Ran `wiki-maintain.sh` (03:01:09 → 03:01:10 AWST): step 1 regenerated `systems/skills-index.md`
|
||
(**346 skills / 70 categories**), step 2 wrote `raw/stocktake/freshness-20261014.txt`
|
||
(**48 stale / 38 fresh / 7 no-date of 93 pages**), step 3 skipped the Quartz rebuild again
|
||
(no `quartz.config.ts`, no `npx`) so `public/` is untouched — the known carried item. **Finding 1 — a second, unclustered Proxmox hypervisor exists: `m93p` (`100.104.51.98`), PVE 9.2.0
|
||
(kernel `7.0.14-12-pve`).** `pvecm nodes` is empty on both `pve` and `m93p` — no cluster — and
|
||
guest IDs **collide** between them (`m93p` CT221 is gitea; `pve` CT103 is cliproxyapi; CT100 and
|
||
CT245 exist on both lists in one form or another). It surfaced because a tailnet-wide scan for the
|
||
hypervisor port `:8006` returned only `m93p` and `pve`. `m93p` had been sitting in the Tailscale
|
||
node list for weeks as an unremarkable `online` peer, with no wiki page recording it as a
|
||
hypervisor. Live guests recorded from a single `pct list` / `qm list` pass. **Finding 2 — four guests the wiki placed on `pve` live on `m93p`.** CT221 (gitea), CT241 → CT1241
|
||
(ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`**,
|
||
yet every one of the four services answers. `/etc/pve/lxc/` on `pve` contains only `260.conf.bak`
|
||
and no vzdump task log references CT 221/241/245/260 — so they were not deleted, **they were never
|
||
on this host**. The 2026-10-11 inventory was wrong and every audit since carried it forward. **Finding 3 — the tailnet inventory had drifted badly.** The page listed **30** nodes; live
|
||
`tailscale status` reports **45** non-funnel nodes (plus 23 `funnel-ingress-node` IPv6 peers, now
|
||
noted and deliberately excluded). Fifteen nodes were missing, including `m93p`, `pve`,
|
||
`omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`,
|
||
`omarchybook`, `sftpgo`, `rclone`, `omni` and `runtipi`. Several states were wrong too
|
||
(`anthonys-mac-mini` is online on a different IP; `omarchy-macmini` / `omarchy-quattro` are not
|
||
live peers; `anthonys-iphone` is online, not active). The CT460 **serve** routes re-verified
|
||
**exactly** as recorded — 13 routes plus `:8443/webhook` Funnel and `:9443` tailnet-only — so that
|
||
table needed no change. **Finding 4 — Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service`
|
||
now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with
|
||
16 tasks, active since 2026-10-13 23:42:55. The containment drop-in held; the workload was the
|
||
fault, as diagnosed last cycle. **Also verified, unchanged:** Mattermost CT255 stopped (peer offline 24d); VM 282
|
||
`ubuntu-server-xfce` is now **running**, not stopped as recorded. **Cron (verified from `executions.db` / `jobs.json`, 03:05):** 2026-10-13 = **168 completed /
|
||
1 failed** (the 2026-10-14 audit's own row finalised at 168); 2026-10-14 in progress **23
|
||
completed / 0 failed** with the 03:00 stocktake `1a8e90ea2271` still `running` at probe time. Open
|
||
incidents **3** — `1a8e90_d11698bd7275` (detected, 10-13 opencode 404), `0e753e_e7f8194a7d06`
|
||
(detected since 10-01, next slot 2026-11-01), `1b23c0_c35703f637b8` (alerted since 10-09, next slot
|
||
Fri 2026-10-16 08:00). Non-clean `last_status`: **4** (unchanged). Fleet **29 jobs — 26 enabled /
|
||
3 disabled**. **Pages touched:** `infrastructure/proxmox.md` (two-host rewrite, m93p inventory, four-entry
|
||
correction, storage, VM 282), `infrastructure/tailscale.md` (45-node inventory + funnel-peer
|
||
exclusion note, frontmatter), `infrastructure/network-map.md` (m93p entry, frontmatter),
|
||
`current-state.md` (new Recent Changes block), `index.md` (proxmox/tailscale blurbs + header
|
||
warning), `systems/skills-index.md` (stocktake artifact), `raw/stocktake/freshness-20261014.txt`
|
||
(new), `log.md`. **Not verified, left as stale:** the remaining 46 stale pages were not probed against a live
|
||
source, so they stay flagged rather than rewritten. Not re-verified this pass and still
|
||
single-host in framing: [[systems/omniroute]], [[infrastructure/hosts]], [[systems/model-providers]],
|
||
[[infrastructure/docker-services]] — `m93p` also runs its own `omniroute` (CT101), so the
|
||
OmniRoute topology needs a dedicated pass before anyone relies on the current four-node story. No secrets written; host names, IPs, VMIDs and job/incident IDs only. ---
|
||
|
||
## [2026-10-16] audit | nightly review of 2026-10-14 → 2026-10-15 (two days — the 10-14 audit died inside the same outage) — a 34-hour fleet blackout from one OmniRoute node
|
||
|
||
Ran 2026-10-16 00:0x–00:3x AWST. **Scope is two windows, not one:** the 2026-10-14 23:59 Wiki Audit
|
||
(`cbfe434a3b5e`) failed on the fault below, so the 10-14 window was never covered. **Finding: fleet-wide cron blackout, 2026-10-14 08:30:27 → 2026-10-15 18:07 AWST (~34 h).**
|
||
|
||
- **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against
|
||
`http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (confirmed in
|
||
`config.yaml`: custom provider `name: omni-101`, `base_url: http://100.78.63.20:20128`). It surfaced as
|
||
`openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` — the exact string
|
||
recorded on every failing execution row. 328 `Errno 111` lines in the live `errors.log`, 504 across the
|
||
rotations. - **Impact.** 10-14 = **160 completed / 12 failed**; 10-15 = **151 completed / 24 failed** (+1 in flight). Against 10-12 `175/0`, 10-13 `167/1`. **36 failure rows across 13 jobs**, but only **24 are genuine
|
||
model-call failures** on **12 jobs** — the other 12 rows are the Mnemosyne Watchdog reddening on a log
|
||
sentinel (symptom, not fault). 10-15 is the worst cron day since 2026-10-09 (15). - **The finding that matters most — there is no failover.** Verified from live `jobs.json`: **all 18
|
||
enabled `agent`-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6,
|
||
`deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600, `omniroute:20128`),
|
||
sits on `e530c89f2be0` — which is **`no_agent: true`**; `d40085da631c` is disabled. Completing runs inside
|
||
the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`**. So **zero model
|
||
calls completed in 34 hours**, and the fleet has no job that could even report the second node's health. - **Correction made during this audit.** A first pass read Evening Stand-Down Briefing's clean 18:45 runs
|
||
(both days) as evidence CT600 stayed healthy. Wrong: it is `no_agent` and never calls a model. Recorded as
|
||
a standing rule — *never read a `no_agent` job's green row as provider-health evidence* — in the incident
|
||
page and in [[systems/scheduled-tasks]]. - **User-visible cost: silence.** Digests never delivered: Tech & AI, Daily Marketing Job Radar, Political
|
||
News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick. The **weekly Perth
|
||
New Restaurant was missed outright** (next slot 10-22). - **Recovery, verified live (not assumed).** `1b31187397a5` completed **10-15 22:01:56** (streak 4 → 0); `96eb95ce12f2` **10-16 00:01:36**; `b63ed5817212` **10-16 00:02:29**; `ea79a2107e36` **10-16 00:09:48**. Probes 2026-10-16 00:1x: `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth
|
||
required); same on `omniroute:20128` / `100.88.81.19`. - **Fleet bookkeeping.** **Non-clean `last_status`: 3 → 12** (9 new: `08eb86f02d37`, `ecc1fc13655a`,
|
||
`dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`,
|
||
`cbfe434a3b5e`; three cleared in the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36`). **Open incidents 3 → 12** (9 `alerted` / 3 `detected`). Long-standing and pre-storm: `1b23c042e17d`
|
||
(Fri 08:00), `0e753e9720fd` (1st 09:00), `066bb5191280` (`delivery_failed`, dead Telegram thread 34795). - **Mnemosyne.** Both monitors sit on the failing node, so **no coverage figures were refreshed for ~34 h**; last explicit read remains 2026-10-12 12:02 (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %). Script-only Mnemosyne jobs were unaffected. - **Self-healing gap left alone deliberately.** The 10-15 03:00 stocktake failure left
|
||
`raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14. Its next
|
||
slot (10-16 03:00) closes the gap, so nothing was regenerated by hand — regenerating would race the
|
||
scheduled run. - **Sessions.** 36 sessions across the two days, **all `cron`, zero interactive** — fourth consecutive
|
||
window with no interactive session. **Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (**new**, 9,303 B); `systems/scheduled-tasks.md` (frontmatter, **Count** current-state bullet, new Re-verified block); `systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
|
||
blast-radius note on the two monitors); `current-state.md` (frontmatter + new top section, `Updated:` line); `index.md` (`Last updated`, new incidents entry, scheduled-tasks bullet); `log.md` (this entry). **Nothing removed or archived** — no retired job, no obsolete system, so `index.md` structure gains one page. **Durable fix still open (needs Anthony's call):** the single-node topology. All 18 model-calling jobs on
|
||
one OmniRoute node means this recurs on the next `omniroute-1` blip — as it did on 2026-10-09. Options are
|
||
a second provider pin on a subset of jobs, or router-side failover. **Not applied** — that is a routing
|
||
decision, not an audit one. No secrets written; host names, IPs, VMIDs and job/incident IDs only. ## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
|
||
|
||
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
|
||
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
|
||
consecutive window with no interactive session, so every finding below is from live system state, not
|
||
conversation. ### Summary
|
||
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
|
||
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
|
||
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
|
||
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
|
||
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
|
||
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
|
||
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new). - **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
|
||
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
|
||
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
|
||
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
|
||
incident. **No change was made on the node**; nothing was restarted or repinned. - **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
|
||
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
|
||
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
|
||
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck. - **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
|
||
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
|
||
sentinel. Read failures per job, not per row. - **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
|
||
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16; 10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
|
||
hand-run would race the scheduled one. - **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
|
||
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
|
||
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
|
||
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
|
||
token is Anthony's call, and the secret is not written here. - **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
|
||
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
|
||
rows are typed `script`), so `failure_type` still cannot be used to triage. - **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
|
||
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
|
||
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
|
||
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
|
||
threshold, so no reindex is due. **Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry); `systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
|
||
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section); `index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note); `log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window. **Bridge housekeeping (verified 2026-10-18):** every page this run touched was byte-identical between the
|
||
authoring tree (`/home/hermes/workspace/agent-estate-wiki`) and the vault copy
|
||
(`/home/hermes/wiki/agent-estate-wiki`, pages nested under `wiki/`) *before* the edits, so the seven files
|
||
were written to the authoring tree first, committed and pushed (`4137dd6`), then copied into the vault —
|
||
sha256 matched on all seven. The vault-only pages (`wiki/systems/ignis.md`, `wiki/systems/nextcloud.md`)
|
||
and the vault's own `wiki/raw/stocktake/` series were not touched; nothing was deleted and no one-way sync
|
||
was run. **Drift found, not fixed:** `incidents/2026-09-02-sftpgo-outage.md` exists only in the authoring
|
||
tree — 17 incident pages canonical vs 16 in the vault — so the vault is still missing that one page. **Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
|
||
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
|
||
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
|
||
an audit one. No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only. ## [2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST
|
||
|
||
**Window verdict: one failure all day, and it was a failure mode the wiki had never recorded.** Fleet
|
||
**28 jobs (25 enabled / 3 disabled)**; `10-18 = 168 completed / 1 failed` against `10-17 = 163 / 5`; `10-19` so far `5 / 0` plus this audit. **11 sessions in the window, all `cron` — no interactive
|
||
sessions**, so no user decisions arrived to document; every finding below is machine-observed. - **A model stream repeated itself and Hermes delivered the placeholder to Telegram.** `ecc1fc13655a`
|
||
(FreshRSS Midday, 12:00) wrote *"Your request was not processed. Send it again if you still want me to
|
||
carry it out."*, **delivered it** (`delivery_outcome: delivered`, 12:08:11) and then died at 12:08:25 on
|
||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
|
||
degenerate response.` New page → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The open
|
||
question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue)
|
||
instead of substituting text on the job's normal route?
|
||
- **Mattermost: cause found — CT255 is stopped**, superseding the 10-11 entry's "cause not determined". Reconnect loop unchanged (`attempt 471` at 00:14:07, `connect timed out after 30s`, 5-minute cadence),
|
||
peer offline **29d**, and the host answers neither TCP `:8065` nor ICMP. Platform config still says
|
||
`enabled: true`, so the loop will keep logging until the container is started or the block retired. - **New page: [[systems/push-notifications]] — the bridge is broken and has been for as long as the logs
|
||
go back.** `status.json`: `connected: false`, `error: …connection refused (nothing listens there on
|
||
this machine)`; the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which
|
||
`/etc/hosts` resolves to **`127.0.1.1`** where nothing serves 443, while the gateway itself is healthy
|
||
on `0.0.0.0:9119` (`/` → 302) and Serve still maps the name to it. **23,469** occurrences of that string
|
||
across `agent.log(.1-3)` + `errors.log(.1-2)`. Fix documented, **not applied**. - **Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.** `working_memory`
|
||
had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18
|
||
03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in
|
||
the page); 1 unresolved conflict. The 24 h threshold in `mnemosyne-health-check.sh:92` will keep
|
||
flagging idle weekends. - **Stocktake landed and closed the two-day hole:** `1a8e90ea2271` exited 0 on 10-18 03:00,
|
||
`freshness-20261018.txt` written, [[systems/skills-index]] regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (`dd53e5f`). - **Carried forward, still open:** the GitHub Trending Radar delivery token (`getMe` → 401, verified
|
||
10-18) — its next exercise is **Mon 2026-10-19 09:00**; the single-node `custom:omni-101` topology with
|
||
no model-call failover; and the vault drift below. **Pages touched:** `incidents/2026-10-18-degenerate-stream-canned-substitute.md` (**new**); `systems/push-notifications.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet +
|
||
FreshRSS row + totals); `systems/messaging-integrations.md` (frontmatter + new Mattermost section); `systems/mnemosyne.md` (frontmatter + new tier/health block); `infrastructure/tailscale.md` (frontmatter +
|
||
inventory header + three missing peers + `mattermost` age); `infrastructure/proxmox.md` (CT255 age); `current-state.md`; `index.md`; `log.md`. **Nothing removed or archived** — no page became obsolete and no
|
||
one-way sync was run. **Bridge housekeeping + vault drift (verified 2026-10-19 00:2x).** Of the ten files this run touched, six
|
||
were byte-identical between the authoring tree and the vault copy *before* the edits; two more
|
||
(`infrastructure/tailscale.md`, `infrastructure/proxmox.md`) differed only because the vault holds
|
||
**older** versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and
|
||
proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (`wiki/`
|
||
nesting) with sha256 verified per file. **Vault drift measured:** against authoring `HEAD`, the vault holds **74 identical pages, 11 stale** and
|
||
**12 missing**. Stale: `infrastructure/network-map.md`, `infrastructure/tailscale.md`,
|
||
`infrastructure/proxmox.md`, `infrastructure/docker-services.md`, `systems/skills-index.md`,
|
||
`systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`,
|
||
`systems/obsidian.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md`. Missing:
|
||
`incidents/2026-09-02-sftpgo-outage.md` (flagged by the 10-18 audit and still absent), the seven
|
||
`agents/*` pages, `reports/communities-wa-social-media-audit-2026.md`, `CONTRIBUTING.md`, plus this run's
|
||
two new pages (copied immediately after). The vault repo's own HEAD is still `9bacd8e mirror(2026-10-16)`
|
||
with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no remote**, so it cannot be
|
||
pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here. No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only. ## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19)
|
||
|
||
- **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to
|
||
`status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 —
|
||
**9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7
|
||
(2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1. - **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being
|
||
carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong
|
||
(`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36`
|
||
is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**),
|
||
`08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**,
|
||
not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1
|
||
note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now
|
||
flagged in the page for confirmation rather than silently overwritten. - **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`),
|
||
not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL`
|
||
count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like
|
||
with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24. - **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from
|
||
`jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet,
|
||
non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected),
|
||
`systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section),
|
||
`index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`. - **Nothing removed or archived** — no page became obsolete and no one-way sync was run. - **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no
|
||
interactive sessions, so there were no user-facing decisions to record. - No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only. - **Vault drift (measured — `sha256` per file against `/home/hermes/wiki/agent-estate-wiki`):** 76 identical /
|
||
15 stale / 0 genuinely missing across the ~91 content pages the vault mirrors. (A raw diff reports 512
|
||
"missing", but those are build artefacts under `public/`, `package*.json`, `serve.py` and `raw/stocktake/`,
|
||
which the vault deliberately does not carry.) **All six pages touched by this audit were copied in with
|
||
per-file `sha256` verification.** The stale pages left unmirrored —
|
||
`infrastructure/docker-services.md`, `infrastructure/network-map.md`, `systems/current-profile.md`,
|
||
`systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`, `systems/obsidian.md`,
|
||
`systems/skills-index.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md` — are authoring-tree edits
|
||
made from *other* agents' sessions; **a full vault mirror pass remains the open call.**
|
||
- **Not re-run this audit:** the `public/` Quartz rebuild and `wiki-maintain.sh`. Authoring commits
|
||
`95fd7d3` → `766882e` → `8da6ddf` → `0248698`, pushed to Gitea `Anthony/agent-estate-wiki` and confirmed at the remote. ### 2026-10-21 — nightly wiki audit (cron `cbfe434a3b5e`, window 2026-10-20)
|
||
|
||
- **A clean day — and a failure this ledger cannot see.** 2026-10-20 closed **168 `completed` / 0 `failed`**
|
||
(plus 1 `unknown`), the **first zero-failure day since 2026-10-12**, ending a seven-day run. The `omni-101` 503
|
||
fault did **not** recur (last failure row 2026-10-19), so
|
||
[[incidents/2026-10-16-omniroute-503-resource-pressure]] returns to **`monitoring`**. **Zero incident rows
|
||
opened, 4 closed → unclosed 13 → 9** (8 `alerted` / 1 `detected`). - **New page: [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]].** The gateway died with no exit path at
|
||
**2026-10-20 09:09:01** (last heartbeat 08:50:28, `pid 2496376`, `suspected_oom=False`), orphaning the 08:30
|
||
**Gmail Action Digest** run (`08eb86f02d37`, claimed 08:31:03) as **`unknown`** — *"owner exited before a
|
||
durable terminal state"*. No delivery row for the slot, **no incident row created**, and the job table still
|
||
shows the *previous* slot's 503. First recorded orphan; first unclean exit since 2026-09-21 (20 unclean exits
|
||
since 2026-08-01, 5 of them clustered on 2026-08-14/15). - **New page: [[incidents/2026-08-08-gitea-config-backup-push-dead]].** `e27da6c34bbb` reads `ok` and its
|
||
executions read `completed` nightly while **nothing has reached Gitea since 2026-08-07 11:00:47** — **75
|
||
unpushed commits**, `fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net'`. Second instance
|
||
of the "green row, dead job" pattern after the GitHub Trending Radar token revocation. - **Corrected the 2026-10-18 canned-substitute page.** Its "second surface" section attributed the
|
||
interactive-session substitution to an `HTTP 404` Model-not-found; the retained log evidence is
|
||
`Rebuilt-message restart limit (3) exceeded` → `reason=rebuilt_restart_limit_exceeded … api_calls=1/150
|
||
tool_turns=0 response_len=0`. The page now carries the verified trigger and flags the unresolved model
|
||
disagreement (session pin `minimax/minimax-m3.1-flash:free` vs `model=qwen3.5:397b` in the turn-end line). **The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered,
|
||
and is recorded as outstanding in [[current-state]].**
|
||
- **Pages updated:** [[current-state]] (new Recent Changes section), [[systems/scheduled-tasks]] (new count bullet
|
||
— non-clean `last_status` 12 → 9, config-backup row annotated, frontmatter), [[systems/current-profile]] (model
|
||
default moved to `nous` / `meituan/longcat-2.5-preview:free`, no `fallbacks` key, pins unchanged),
|
||
[[infrastructure/backups]] (the scheduled Gitea push has been dead since 2026-08-08),
|
||
[[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to `monitoring` + dated update),
|
||
[[incidents/2026-10-18-degenerate-stream-canned-substitute]] (trigger corrected), `index.md`, this log. - **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **20 of 28 pin OmniRoute**; the **8 unpinned jobs
|
||
moved upstream to `nous`** with the global default change — the fleet now runs two upstream paths. **2
|
||
interactive sessions in the window** (`talaria-ios` tax-return work; the `desktop` turn served the canned
|
||
non-answer) — the first non-cron sessions after weeks of cron-only windows. - **Nothing removed or archived** — no page became obsolete, and no one-way sync was run. - **No secrets written.** The gateway token pasted in the 2026-10-20 desktop session was **not** transcribed into
|
||
any page; both the incident and current-state entries describe the request without the credential. Host names,
|
||
IPs, job/incident IDs, ports and probe status codes only. ## [2026-10-21] stocktake | Scheduled wiki maintenance (cron `1a8e90ea2271`, 03:00 AWST)
|
||
- **Maintenance script ran clean** — `wiki-maintain.sh` completed all three steps: skills-index regenerated
|
||
(**346 → 347 skills**; new `australian-tax-return-review` under Finance), freshness audit written
|
||
(`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed to `public/`,
|
||
`hermes-wiki-serve.service` restarted. - **[[infrastructure/tailscale]] — node table rebuilt from live `tailscale status --json`.** The 2026-10-19
|
||
table had drifted again: **`octop` (100.92.163.122) and `springroll` (100.100.153.125) were missing entirely**,
|
||
`openmausbot` / `openmausbot-vm` were described in prose but absent from the table body, `omarchymac` had gone
|
||
offline (online → idle/offline 3d), and every offline age was stale. Table regenerated keyed on **DNS name** rather
|
||
than the JSON `HostName` — the latter is `ct260` for the `freshrss` node, and **both OmniRoute nodes share the
|
||
hostname `omniroute`**, which silently collides. Now **51 non-funnel nodes** (50 IPv4 peers + `hermes`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98):
|
||
CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running. - **[[reports/uber-eats]] — two missing monthly reports recovered.** The generator
|
||
(`~/.hermes/cron/scripts/generate-uber-eats-report.py`) wrote only to `~/wiki/reports/uber-eats/` (the `~/wiki`
|
||
repo), **never to the canonical `agent-estate-wiki` tree**, so the 2026-08 and 2026-09 reports were missing from
|
||
the published wiki. **Root cause fixed:** the script now writes to `~/workspace/agent-estate-wiki/reports/uber-eats/`
|
||
and mirrors to the legacy tree. It also **now emits YAML frontmatter** — without it the pages landed in the
|
||
"no updated DATE" bucket of the freshness audit and Quartz warned it could not read the page date. Both missing
|
||
months copied in and verified against `~/.hermes/uber_eats_report.json` (2026-09: $51.29 / 2 orders; 2026-08:
|
||
$211.16 / 7 orders). The **2026-08 previous-month column was corrected** — the original run had no comparison
|
||
payload and wrote $0.00 against 2026-07; it now shows $264.51 / 11 orders, a -$53.35 change. - **[[incidents/2026-08-15-skills-index-drift]] and [[incidents/2026-09-02-sftpgo-outage]] re-verified against live
|
||
state** and annotated. The skills-drift page named a `regenerate-skills-index.sh` that **no longer exists on disk**
|
||
(regeneration now lives inside `wiki-maintain.sh` step 1/3); the SFTPGo page confirmed **CT299 running** and the
|
||
health endpoint returning **HTTP 401** (auth required, service up) — no recurrence. - **Still stale, left alone deliberately:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source
|
||
document `/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists on disk**,
|
||
so the audit baseline cannot be re-verified against a live source. Flagged, not edited. - **Fleet unchanged** — 28 jobs / 25 enabled. This job (`1a8e90ea2271`) is `active`, `last_status: ok`,
|
||
`model: free-stack` / `provider: custom:omni-101`, next run 2026-10-22 03:00. - **No secrets written.** Only host names, IPs, container IDs, job IDs, port numbers and HTTP status codes. ## [2026-10-21] audit | Nightly wiki audit — window 2026-10-21 00:00 → 23:59 AWST: Hermes updated to 0.22.3 with its snapshot skipped, and a third failure signature from the `omni-101` pin
|
||
|
||
- **New page: [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].** The **Daily Marketing Job Radar**
|
||
(`dca8482e4f76`, 07:00 weekday, `custom:omni-101` / `smart-route`) failed **2026-10-21 07:40:48 after
|
||
40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||
filters reset after 19m 20s` — a string this fleet has never produced before, and **not** the 503
|
||
resource-pressure error (zero such rows exist for 10-21; last 2026-10-19 09:02). Session
|
||
`cron_dca8482e4f76_20261021_070022` wrote no output (`cron_incomplete_no_output`, 82 msgs / 47 tool calls); incident `dca848_014affeb6954` is **`alerted` and unclosed**; it was the day's only failure
|
||
(171 `completed` / 1 `failed`). - **Infrastructure: Hermes Agent updated to `0.22.3`** — `hermes update` ran **22:47:25 → 22:57:38 AWST**
|
||
(action `9eed3a99e799347c691ea52a72e122a1`), pre `0.22.2` / `d923a5eb…` → commit `5b147ab036…` (detached),
|
||
stamp built 2026-10-21T14:57:38Z. **Stage `snapshot` = `skipped`** (`disabled by
|
||
updates.pre_update_backup (mode: off)`), so the update had no snapshot and the off-box copy path is still
|
||
dead. Gateway restarted via systemd (pid `868087`, `phase: running`, heartbeat 00:13 AWST), dashboard
|
||
respawned by argv (`:9119`, pid `869852`). Advisories printed, neither actioned: rebuild the search index
|
||
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver…* (no completion line in the log). The intermediate `d923a5eb` rebuild was never recorded on the page, so the previously-noted
|
||
`v0.21.5+6169.gab86a20` (2026-10-04) was already stale — corrected. - **Pages updated:** `systems/hermes-agent.md` (version/upstream bullets rewritten, frontmatter dated),
|
||
`systems/scheduled-tasks.md` (new current-count bullet; 10-21 bullet marked superseded),
|
||
`systems/omniroute.md` (new 2026-10-21 section: three signatures, one pin),
|
||
`incidents/2026-10-16-omniroute-503-resource-pressure.md` (window-close update; stays `monitoring`),
|
||
`incidents/2026-08-08-gitea-config-backup-push-dead.md` (2026-10-21 run: backlog 75 → **76**,
|
||
`message_id 8449`), `infrastructure/backups.md` (76 commits; `pre_update_backup: false` now has a
|
||
concrete exercised example), `current-state.md` (new dated section), `index.md` (new incident entry,
|
||
scheduled-tasks summary, date). - **Ledger facts as recorded this window:** fleet **28 jobs / 25 enabled / 3 disabled**; non-clean
|
||
`last_status` **9** (7 `error` / 2 `delivery_failed`) with one swap — `08eb86f02d37` out (its 08:30 slot ran
|
||
clean and `08eb86_472c7b609c80` **resolved 08:37:21**), `dca8482e4f76` in; **1 incident opened / 1 closed,
|
||
unclosed stays 9** (8 `alerted` / 1 `detected`); **12 sessions, all `cron`, no interactive sessions**; 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the orphan) — the 10-21 audit's "168 / 0" was taken
|
||
before the 23:59 slot landed, and that correction is recorded in both `systems/scheduled-tasks.md` and
|
||
`current-state.md`. - **No secrets written.** Host names, IPs, job IDs, incident IDs, ports, commit SHAs and HTTP status codes only. ## [2026-10-22] maintenance | Quartz rebuild + deploy and skills-index regeneration (00:17-00:18 AWST)
|
||
|
||
- Ran `~/.hermes/scripts/wiki-maintain.sh` after the audit so the new incident page is actually published,
|
||
rather than waiting for the 03:00 stocktake. - **`systems/skills-index.md` regenerated from the live tree:** **348 skills** (was 347) across 74 top-level
|
||
directories; section order now follows the script's category-priority list. Auto-generated — no hand edits. - **Freshness report `raw/stocktake/freshness-20261022.txt`:** **1 stale / 95 fresh / 6 no-date of 102 pages**. The only stale page is `reports/communities-wa-social-media-audit-2026.md` (55 d). - **Quartz build + deploy:** 273 HTML files into `public/`; `hermes-wiki-serve.service` restarted 00:18:05 AWST
|
||
(the build wipes and recreates `public/`, so the running server always needs the restart). - **Verified live:** `http://127.0.0.1:9120/incidents/2026-10-21-omniroute-pre-dispatch-filter-skip` → 200, and
|
||
all four of its wikilinks resolve (`systems/omniroute`, `systems/scheduled-tasks`, `systems/hermes-agent`,
|
||
`incidents/2026-10-16-omniroute-503-resource-pressure`); the new entry appears twice on the served index. ## [2026-10-23] audit | Nightly wiki audit — clean cron day, five incidents closed, skills-index drift reopened (00:1x AWST)
|
||
|
||
- **`systems/scheduled-tasks.md`** — new `## Count` bullet for the 2026-10-22 window: **172 `completed` / 0 `failed`**
|
||
(+ this audit `running`), zero failures for the second time in three days; fleet unchanged **28 jobs — 25 enabled /
|
||
3 disabled**; **five incidents closed in-window and none opened**, so unclosed rows fall **9 → 4**; non-clean
|
||
`last_status` falls **9 → 5**; **27 sessions** (15 `cron`, 3 `subagent`, 6 `desktop`, 2 `acp`, 1 `oneshot`). Previous bullet marked superseded; `updated`/`verified_on` bumped to 2026-10-23, `review_after` to 2026-10-30. - **`current-state.md`** — new `Recent Changes (2026-10-23)` block at the top: the 172 / 0 window, incidents
|
||
**9 → 4** unclosed and non-clean `last_status` **9 → 5**, the 10-21 recount (168 / 1), the stocktake
|
||
no-commit anomaly, the skills-index root fix, the Quartz deploy, and the session mix; `updated`/`verified_on`
|
||
bumped to 2026-10-23, `review_after` to 2026-10-30. - **Correction recorded:** a DB recount (by both `started_at` and `claimed_at`, AWST) puts **2026-10-21 at 168
|
||
`completed` / 1 `failed`**, three below the 171 / 1 this log recorded on 2026-10-22. Retention does not explain it
|
||
(floor 2026-10-08); the delta is written into the page rather than silently changed. - **`incidents/2026-08-15-skills-index-drift.md` reopened** (`status: resolved` → `active`, `reopened_on:
|
||
2026-10-23`). Evidence: the 2026-10-22 stocktake (`1a8e90ea2271`) ran **03:00:28 → 04:38:19 (97 m 58 s, vs
|
||
16 m 01 s on 10-21)**, finished `completed` with delivery `suppressed`, and **committed nothing** — its
|
||
regenerated `systems/skills-index.md` (mtime 03:00, `## .Archive (41)` section, 347 total) was still
|
||
uncommitted and its diagnosed fixes unapplied. Two generators produce two shapes for the same page: the repo's
|
||
`wiki-maintain.sh` (`find`-based, 347 with `.Archive`) and the stale `~/.hermes/scripts/wiki-maintain.sh`
|
||
(`rglob`-based, 348 without). Live counts verified: filesystem **348**, with symlinks **376**, minus `.archive/`
|
||
(41) and `apple/` (4) = **331**, loader catalogue **329 enabled** (34 hub-installed / 52 builtin / 243 local). - **Root-cause fix applied:** the repo's `wiki-maintain.sh` now prunes `*/.archive/*` from the skill scan (the
|
||
loader does not expose archived skills), prints the archived count as an explicit exclusion, and names
|
||
`hermes skills list` as the authoritative loadable figure. **`systems/skills-index.md` regenerated: 306 skills
|
||
across 69 categories** (was 347 counting the parked archive). Script fix and regenerated page committed together. - **Quartz rebuild + deploy:** 102 files parsed, **602 files emitted to `public/`**, deployed, and
|
||
`hermes-wiki-serve.service` restarted 00:14:22 AWST. - **Freshness report `raw/stocktake/freshness-20261023.txt`:** **2 stale / 94 fresh / 6 no-date of 102 pages** —
|
||
the long-standing `reports/communities-wa-social-media-audit-2026.md` (56 d) and, newly over the threshold,
|
||
`systems/model-providers.md` (22 d, updated 2026-10-01). The latter needs a review pass against the live
|
||
OmniRoute catalogue before its next `verified_on`. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. ## [2026-10-24] audit | Nightly wiki audit — window 2026-10-23 00:00 → 2026-10-24 00:3x AWST
|
||
|
||
- **New page:** `systems/conduit.md` — the Conduit dashboard plugin + iOS app (voice calls, cards,
|
||
push). Plugin tree `~/.hermes/plugins/conduit_push`, version `0.64.0` (HEAD `eaa14be`, no tags
|
||
upstream), remote `kaishi00/hermes-conduit-notifier`; gateway on `:9119`; one registered device
|
||
(iPhone, app `2.0.0`); state/DB files and sizes recorded. Where the wiki previously had no page for
|
||
the phone surface at all. - **New incident:** [[incidents/2026-10-22-conduit-memory-start-timeout]] — three
|
||
`Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call` warnings
|
||
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) from a hard-coded 10 s `MEMORY_TIMEOUT_S` in the
|
||
plugin against Mnemosyne's 10–45 s cold start; calls ran without recall. Patched in the working tree
|
||
to a 60 s default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that degrades
|
||
instead of aborting; the dashboard restarted 18:54:43 and **no abort line exists after it** (two
|
||
calls closed cleanly at 00:03:23 and 00:06:57 on 10-24). **The patch is uncommitted.**
|
||
- **`incidents/2026-10-09-cron-provider-timeout-storm.md` → `status: resolved`.** Its last two rows
|
||
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`, both on `1b23c042e17d`) closed **2026-10-23 08:04**
|
||
when that weekly job's Friday slot ran clean; the job reads `ok` again. **All 12 rows closed** —
|
||
this incident no longer contributes an unclosed row anywhere. - **Cron ledger:** window **2026-10-23 closed 169 `completed` / 0 `failed`** (+ this audit `running`) —
|
||
the third zero-failure day in four; **2026-10-22 settled at 173 / 0**. Fleet **28 jobs (25 enabled /
|
||
3 disabled)**, unchanged for a sixth audit. **Unclosed incident rows 4 → 2** (`c0a9fa_764558b5ae83`
|
||
`alerted` since 10-19 — GitHub Trending Radar revoked token, next slot Mon 10-26 09:00; and
|
||
`0e753e_e7f8194a7d06` `detected` since 10-01, next slot 11-01). **Non-clean `last_status` 5 → 4**
|
||
(`c0a9fa314542`, `0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). Pins unchanged (11 `omni-101` /
|
||
`smart-route`, 8 unpinned, 6 `omni-101` / `free-stack`, 2 `custom:omni` / `smart-route`, 1
|
||
`deepseek-paid-smart` = this audit). **14 sessions:** 13 `cron` + 1 `desktop` (18:26:12, the Conduit
|
||
phone-call session that exposed the fault above). - **`systems/mnemosyne.md`:** new block recording the 10–45 s cold start and the consumer-timeout rule
|
||
(a 10 s budget aborts a voice call; consumers need ≥60 s or must degrade to "no recall"). The cold
|
||
start itself was **not** optimised — only the budget was raised. - **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-24`
|
||
section), `current-state.md` (new Recent Changes block; 10-23 block marked superseded), `index.md`
|
||
(date, new `systems/conduit` and incident lines, refresh of the scheduled-tasks / skills-index /
|
||
storm blurbs), and **frontmatter added** to `incidents/2026-07-29-mnemosyne-relay.md` and
|
||
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md` — both sat outside the SCHEMA
|
||
(rule 4: every page has frontmatter). - **Freshness:** 102 pages scanned at the last stocktake; **3 carry a `verified_on` older than 21 days**
|
||
— `infrastructure/hosts.md`, `runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`
|
||
(all 94 d, unchanged). **Nothing removed or archived** this cycle: no job retired, no system
|
||
obsolete, no decision reversed in the window. - **Note (hygiene):** a second, older copy of this wiki sits at `/home/hermes/wiki/agent-estate-wiki`
|
||
(101 pages, 3 incidents, no git remote, last commit `c798304` "mirror(2026-10-22 maintenance)"). It is **not** served — `hermes-wiki-serve.service` serves `public/` from *this* repo — so it is a
|
||
stale duplicate that will drift. Nothing in the audit depends on it; flagging it rather than
|
||
syncing it, since no script owns it. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and HTTP status codes only. ## [2026-10-25] audit | Nightly wiki audit — window 2026-10-24 00:00 → 2026-10-25 00:5x AWST: a third consecutive clean cron day, and a Conduit call request the agent refused
|
||
|
||
- **Window 2026-10-24: 167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled
|
||
at 170 / 0** (169 at the 10-24 audit). Third consecutive clean day (10-22 → 10-24). Fleet **28 jobs
|
||
(25 enabled / 3 disabled)**, **4 non-clean `last_status`**, **2 unclosed incident rows**
|
||
(`c0a9fa_764558b5ae83` `alerted` since 10-19; `0e753e_e7f8194a7d06` `detected` since 10-01) —
|
||
**nothing opened, nothing closed in-window**. Pins unchanged. **14 sessions: 11 `cron` + 3
|
||
`desktop`.**
|
||
- **New incident page: `incidents/2026-10-24-conduit-call-request-refused.md`.** Anthony asked to be
|
||
phoned three times on 2026-10-24 12:53–12:56 (`Call me now to test calls.` → `Yes. You do` →
|
||
`CAL ME`); every reply asserted the agent cannot make calls, with **0 tool calls** in the session. `conduit_call_user` exists, is registered for every session, is discoverable via `tool_search`, and
|
||
has a bundled skill (`calling-the-user`); the call settings were permissive
|
||
(`enabled` / `when_asked` / `decides` / `alerts` all true). Root cause: **capability asserted
|
||
instead of discovered** — not a missing tool, and not one weak model (the free model's answer was
|
||
repeated after a switch to `smart-route`). - **Correction to `incidents/2026-10-22-conduit-memory-start-timeout.md`:** its closing evidence
|
||
(“two Conduit call sessions closed cleanly 00:03:23 / 00:06:57 on 10-24”) is a **false positive**. `conduit-calls.json → ends` is a ring buffer of **turn-ends that fire a watch**
|
||
(`MAX_RECENT_ENDS = 64`), and its two entries are the **cron** sessions of that night (Mnemosyne
|
||
Embedding Coverage Monitor 00:01, Health Monitor 00:04). `history` — real call records — is
|
||
**empty**, so no call has been placed since the fix and the 60 s budget is **unexercised by a
|
||
call**. The surviving evidence for the fix is only the absence of abort lines after the 18:54:43
|
||
restart. - **`systems/conduit.md` refreshed:** version `0.64.0` / `eaa14be` → **`0.69.0` / `f000bf0` (22
|
||
commits on)**; the gateway is still the 2026-10-23 18:54:43 process, so none of it is live; the
|
||
uncommitted `plugin_api.py` patch is now **five releases behind HEAD**; new section documenting the
|
||
`conduit_call_user` tool, its gate conditions and rate limits; device/state figures re-verified
|
||
(`last_seen` 10-23 19:10:30 tracks phone-status contact; health ingestion ran to 10-24 09:59:59); store **569.9 MB / 1,520,229 events / 3,028 `buckets`** — the 10-24 page's 308 `buckets` / 62
|
||
`windows` was a **narrower grouping**, corrected rather than read as a collapse. - **Also updated:** `systems/scheduled-tasks.md` (new Count bullet + `## Re-verified 2026-10-25`
|
||
section + a date-bucketing note), `current-state.md` (new Recent Changes block; 10-24 block marked
|
||
superseded), `index.md` (date, conduit + scheduled-tasks blurbs, new incident line, corrected 10-22
|
||
line). - **Freshness:** not re-scanned this cycle — the 03:00 stocktake owns that. Carried over from its last
|
||
run: 102 pages scanned, **3 carry a `verified_on` older than 21 days** (`infrastructure/hosts.md`,
|
||
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` — all 94 d, unchanged). They are
|
||
the next cleanup candidates. **Nothing removed or archived** this cycle: no job retired, no system
|
||
obsolete, no decision reversed in the window. - **Method note recorded in `systems/scheduled-tasks.md`:** `executions.db` stores `started_at` with a
|
||
`+08:00` offset and SQLite's `date()` normalises to UTC, so `date(started_at)` moves 00:00–07:59
|
||
AWST rows onto the previous day. Bucket days with `substr(started_at,1,10)`. - **No secrets written.** Job IDs, incident IDs, table names, counts, paths and timestamps only. ## [2026-10-25] stocktake | wiki-maintain.sh + drift correction (tailscale, homepage-dashboard, 2 stale pages)
|
||
- **Script ran clean.** `wiki-maintain.sh` completed all three stages at 03:02–03:03 AWST:
|
||
`systems/skills-index.md` regenerated (302 skills / 69 categories, 45 archived — the count fell
|
||
from 306 because four skills moved to `~/.hermes/skills/.archive/`: `courier-tracking-monitor`,
|
||
`pve-vm-console-recovery`, `model-provider-comparison-research`, `live-application-verification` —
|
||
all four confirmed present in the archive directory), freshness audit written to
|
||
`raw/stocktake/freshness-20261025.txt`, Quartz rebuilt (630 files) and deployed to `public/`,
|
||
`hermes-wiki-serve.service` restarted. Verified the served pages return **200** afterwards. - **Freshness audit at run time:** 2 stale / 99 fresh / 4 no-date of 105 pages. The two stale pages
|
||
were both re-probed live and corrected (see below); after correction the audit reports
|
||
**0 stale**. - **`infrastructure/tailscale.md` — node table rebuilt from live `tailscale status --json`.** The
|
||
2026-10-21 table had drifted again:
|
||
- **`springroll-browser` (100.109.221.91) was missing from the table entirely** — added (online). - **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline**
|
||
(online → idle/offline 3d). `openmausbot`'s m93p LXC (CT1264) is still running but no longer
|
||
has the `tailscale` binary on PATH, so the node itself has stopped connecting. - **Every offline age was stale** and was refreshed: `mattermost` 31d → **35d**, `omni` 64d → **68d**,
|
||
`elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `cloudreve` 88d → **92d**,
|
||
`anthonys-imac` 27d → **31d**, `cachyos` 47d → **51d**, `conduit` 44d → **48d**,
|
||
`freshrss-new` 64d → **68d**, `omarchymac` 3d → **7d**. - `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it offers
|
||
exit-node capability but is not currently serving as one; the old "active; offers exit node"
|
||
wording overstated the live state and was corrected. - `m93p` (100.104.51.98) reads **idle/offline 13m ago** in the peer table, but the host is plainly
|
||
reachable (both `pct list` probes and the SSH session succeeded during this audit), so it is
|
||
recorded as a Tailscale-side liveness artefact, not a hypervisor outage. - Container identities confirmed the authoritative way this run: `tailscale ip -4` executed **inside**
|
||
each candidate LXC on both `pve` (100.96.100.82) and `m93p` (100.104.51.98), because peer names
|
||
and LXC names do not always match. CT209 (cloudreve), CT255 (mattermost), CT555 (elementsynapse)
|
||
and CT601 (omni) confirmed **stopped**; CT600 (omniroute, live OmniRoute) and CT101 (m93p)
|
||
confirmed **running**. - Funnel peer count re-checked: still **23** `funnel-ingress-node` peers with IPv6-only addresses
|
||
(excluded). Total non-funnel nodes 51 → **52** (51 IPv4 peers + `hermes` itself). - **`infrastructure/tailscale.md` + `systems/homepage-dashboard.md` — the CT201 route tables were wrong.**
|
||
Live `pct exec 201 -- tailscale serve status` shows CT201 serving Tailscale on **both `:443` and
|
||
`:8443`**, and the old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead"
|
||
note was **false**: no Traefik container or process exists on CT201 (`docker ps -a` and `ps aux` both
|
||
come back empty for `traefik`), and `ss -ltnp` shows `tailscaled` bound to `100.96.244.39:443`. Corrected routes: `/` → 8085 and `/mgr` → 8084 on **both** ports; `/steel` → 5173 and
|
||
`/steel-api` → 3000 on **`:443` only** (`/steel` on `:8443` returns **502**; on `:443` it returns
|
||
**200**). The old table also routed `:8443` straight to `http://localhost:8082`, which has **no
|
||
listener** — the live target is 8085 (`asset_proxy.py`), which itself proxies to the dead 8082 and
|
||
returns 502. Both pages corrected. The **CT460** route table was checked and matched live exactly
|
||
(12 paths, no drift). - **`reports/communities-wa-social-media-audit-2026.md` (was 58d stale) — re-verified, not drifted.**
|
||
Checked against the live Department of Communities pages: the remit wording, the ten portfolios,
|
||
the **1 July 2025** Housing → Department of Housing and Works restructure, the five-minister
|
||
structure and the WA Seniors Card carve-out all still hold; the Department site was updated
|
||
23 October 2026 and its recent announcements (carers, youth grants, forced adoption) match the
|
||
content pillars recorded here. Audience metrics were **not** re-counted and are left as the
|
||
2026-08-28 point-in-time figures; `review_after: 2026-11-26` stands. - **`incidents/2026-08-26-work-outlook-basic-auth-disabled.md` (was 22d stale) — re-probed, still open.**
|
||
`himalaya envelope list -a work -s 1` reproduces the identical failure (`cannot build IMAP client`
|
||
/ `login not supported`). The tenant still advertises only `XOAuth2` and the account still
|
||
authenticates by password, so no remediation has landed. Status stays `open`; `updated` and
|
||
`verified_on` bumped to the verification date with the probe output recorded on the page. - **`systems/scheduled-tasks.md` checked and left unchanged.** Live `~/.hermes/cron/jobs.json` reads
|
||
28 jobs — 25 enabled / 3 disabled, matching the page. All 28 job IDs are present on the page. Non-clean `last_status` is still **4** (`0e753e9720fd` error, `fad5a86979d4` delivery_failed,
|
||
`066bb5191280` delivery_failed, `c0a9fa314542` error streak 1) and unclosed incident rows are
|
||
still **2** (`0e753e_e7f8194a7d06`, `c0a9fa_764558b5ae83`) — identical to the page's 2026-10-25
|
||
entry. No drift, nothing to change. - **No secrets written.** Job IDs, incident IDs, IPs, counts, paths and timestamps only. ## [2026-10-26] audit | Nightly wiki audit — window 2026-10-25 00:00 → 2026-10-26 00:5x AWST: a runaway digest and a dead Nous credential
|
||
|
||
- **Fleet was quiet but one job was not.** Live `executions.db`: 2026-10-25 settled **172 `completed` / 2 `failed`**; **both failures are `08eb86f02d37` (Gmail Action Digest)**, which also fired **six times** in the day (08:31:14 →
|
||
10:26:37 runaway at **1 h 55 m** vs 91 s / 104 s the two prior days; then 17:19:36, 18:40:06, 19:18:41, 19:30:31,
|
||
19:38:53). New signature on the 19:18:41 fire: **`Cron admission missing: the owner never admitted this fire`**. All six fires — failures included — carry `delivery_outcome: delivered`. **Neither failure created a `cron_incidents`
|
||
row**; the page's open-incident count is unchanged at 2. → new `incidents/2026-10-25-gmail-digest-runaway-and-refire.md`
|
||
- **Root cause of the missing context compression found: the Nous Portal credential is dead.**
|
||
`Nous Portal runtime credentials unavailable: Invalid refresh token` — **1** line 2026-10-22, **3** on 10-24,
|
||
**20** on 10-25 (last 14:53:04, `agent.context_compressor: Failed to generate context summary`), **0** since the
|
||
19:27:13 restart. → new `incidents/2026-10-22-nous-portal-refresh-token-invalid.md`
|
||
- **⚠ The global model default moved BACK to OmniRoute — and the last three audits recorded the stale value.**
|
||
A 2026-10-24 19:06:10 switch note reads *"model was just switched from meituan/longcat-2.5-preview:free to
|
||
`smart-route` via omni-101"*; live config confirms `custom:omni-101` / `smart-route` (`api_mode:
|
||
chat_completions`), `config.yaml` last written 2026-10-25 19:13. A **six-provider top-level `fallback_providers`
|
||
chain is live again** (headed by the dead `nous` credential), so the 2026-10-21 "no fallback chain" record is
|
||
stale too. Consequence: the 8 unpinned jobs are back on OmniRoute → **the fleet is a single upstream path again**. Corrected on `systems/current-profile.md`, `systems/model-providers.md`, `concepts/model-routing.md`,
|
||
`comparisons/model-routers.md`, `systems/omniroute.md`. - **Auxiliary routing captured.** At 2026-10-25 19:13 **all eight** `auxiliary:` slots (approval, compression,
|
||
curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off `nous` to
|
||
**`no-think/freeapi/auto` on `custom:omni-101`** with `free_only: true` and a `freeapi/deepseek-v4-flash` →
|
||
`kg/kilo-auto/free` chain. ⚠ Masking, not fixing — `nous` remains first in `fallback_providers`. New section on
|
||
`systems/model-providers.md`. - **`systems/scheduled-tasks.md` — new 2026-10-26 entry added**; the 2026-10-25 entry marked superseded. Fleet
|
||
unchanged (28 jobs / 25 enabled / 3 disabled, all 28 IDs present); non-clean `last_status` **4 → 5**
|
||
(`08eb86f02d37` joined as `delivery_queued` with an empty `last_error`); pins unchanged. - **`systems/conduit.md` re-probed live.** Version **`0.69.2`** in `plugin.yaml` *and* `dashboard/manifest.json`
|
||
(both), HEAD **`942ddb8`** (2026-10-24 15:35:42, *"Pulse: list candidates, decide in code, journal them (#342)"*),
|
||
**0 behind `origin/main`**, 24 commits on since the `0.64.0` baseline, working tree still carrying the
|
||
uncommitted `dashboard/plugin_api.py` patch. **The dashboard gateway restarted 2026-10-25 18:42:33, so the
|
||
upstream commits are now live** — the previous "not live" note is superseded. `conduit-calls.json → history`
|
||
is still **empty**; `conduit_show` ran once on 10-25 14:37. Status on
|
||
`incidents/2026-10-24-conduit-call-request-refused.md` **stays open** (untested, not fixed). - **`systems/messaging-integrations.md`** — CT255 still `stopped` (Mattermost offline **36 d**, `tx 312 / rx 0`); 268 timeouts on 10-25 vs 264 on 10-24; reconnect counter reset to 54 by the 19:27 restart (per-process, not
|
||
outage duration). - **`systems/hermes-agent.md`** — gateway restart record added (three starts on 2026-10-25: 00:32:20, 18:30:04,
|
||
19:27:13; pid 2476414 up since 19:26:33) plus the credential fault and its restart-resets-counters caveat. - **`index.md`** — header date, the two new incident links, and rewritten `conduit`, `messaging-integrations` and
|
||
`scheduled-tasks` summaries. - **No secrets written.** Job IDs, incident IDs, base URLs, counts, paths and timestamps only. - **Not changed:** `current-state.md` (its dated sections are historical records), and the dead `custom:omni`
|
||
provider entry (still named by 2 jobs — Anthony's call, not the wiki's). ---
|
||
|
||
## 2026-10-26 — Stocktake: 8 stale pages corrected against live state (0 stale remaining)
|
||
|
||
`wiki-maintain.sh` ran clean (302 skills indexed, Quartz rebuilt, 634 files emitted, `hermes-wiki-serve.service`
|
||
restarted). The first freshness report listed **8 stale pages**; each was then **diffed against live state** and
|
||
corrected where reality could be verified. A second script run then rewrote the same-day report with
|
||
**0 stale / 103 fresh / 4 no-date**, so the committed report matches the tree it ships with. - **`infrastructure/hosts.md`** — **CT600 is live, not retired.** Live `pct list` on both hypervisors shows
|
||
**CT600 running** on `pve` (100.88.81.19) and **CT101 running** on `m93p` (100.78.63.20); **CT601 does not
|
||
exist on either host** (`pct config 601` → no such config), so the old "retired → CT601" note was wrong on both
|
||
counts. **CT333** (`clawtest`) moved from "unverified" to verified running. Both hypervisors now carry a
|
||
verified row (`pve` 100.96.100.82 / 17 LXCs; `m93p` 100.104.51.98 / 19 LXCs). Config version **33 → 51**; skills **66 → 302**; cron **"25 active" → 28 (25 enabled / 3 disabled)**; Xvfb display **:102 → :99**
|
||
(`Xvfb :99 -screen 0 1600x1200x24`). The `omni` provider entry was replaced by the two real ones —
|
||
`custom:Omniroute:20128` → `http://omniroute:20128/v1` (6,480 models) and **`custom:omni-101`** →
|
||
`http://100.78.63.20:20128` (**the live default provider**). The authoring tree vs the vault mirror are now
|
||
distinguished by path. - **`agents/hermes-production.md`** — **Tailscale IP corrected: `100.96.100.82` → `100.118.5.51`.** The old value
|
||
is the **PVE hypervisor**, not CT460. Version **v0.21.5 → v0.22.3** (`5b147ab036d6`); stable **v0.22.5** is
|
||
available (`hermes update --check`). The **dead 8082** port claim was removed (no listener; live ports are
|
||
9119 gateway + 8085 webhook). Default provider is **`custom:omni-101`**, and the **fallback chain was rewritten
|
||
from live config** — its **first entry is the dead `nous` credential**, so a fallover tries a broken provider
|
||
first. All **eight `auxiliary:` slots** are `no-think/freeapi/auto` on `custom:omni-101` (moved 2026-10-25
|
||
19:13). Cron **30 (27/3) → 28 (25/3)** with the 5 non-clean `last_status` named. Mattermost still unreachable —
|
||
CT255 `stopped`, peer offline **36 d**. - **`agents/hermes-chaos.md`** — version **v0.22.3**, and **both CT450 and CT460 now run the same build**
|
||
(`5b147ab036d6`); the "**3 commits behind**" claim is superseded (**0 behind `origin/main`**). Default model
|
||
is **`smart-route`** on `custom` → `http://100.78.63.20:20128`, not the old `deepseek-flash-mix` on
|
||
`custom:omniroute`. Cron **"11 paused" → 25 jobs (13 enabled / 12 disabled)**, with **9 names shared with
|
||
CT460** (8 disabled here, `Important-mail monitor` the reverse). - **`infrastructure/tailscale.md`** — node table rebuilt from `tailscale status --json` (keyed on `DNSName`,
|
||
IPv4-filtered) and container identity re-confirmed from **inside** each LXC via
|
||
`pct exec <vmid> -- tailscale ip -4` on both hypervisors. **`octoagent` (100.117.10.68) and
|
||
`omarchymac-1` (100.126.173.115) were missing entirely** and are now listed — **52 → 54 non-funnel nodes**
|
||
(53 live IPv4 peers + `hermes`). `m93p` now reads **online** (it was a stale peer record). **Every offline age
|
||
was refreshed** (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**,
|
||
`ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**,
|
||
`openmausbot` 3d → **4d**). The `omni` row now reads "**CT601 removed from both hypervisors**" rather than
|
||
"CT601 stopped". Serve tables re-probed per host: **CT460 matched live exactly** (12 paths on :443 plus the
|
||
:8443 Funnel and :9443 route); **CT201 still 502s on `/`** (asset proxy → dead 8082) while `/mgr` and `/steel`
|
||
return **200** — the 502 claim was re-confirmed, not merely carried over. - **`systems/memory-backend.md`** — re-verified against live `config.yaml`: added `reflect.disabled_for_cron`,
|
||
`sleep_threshold: 50`, `nudge_interval: 10`, and live totals (**2,350 memories — 842 working / 1,090 episodic /
|
||
418 `memoria_facts`**). Embedding coverage re-measured: **1,899 / 2,350 (80.8 %)** overall, **working 815 / 842
|
||
(96.8 %)** and **episodic 1,084 / 1,090 (99.4 %)** — both tiers **above the 95 % reindex threshold**, so no
|
||
reindex was triggered. - **`runbooks/provider-health.md`** — the step-3 OmniRoute snippet was **broken and raised `IndexError`**: the
|
||
provider was renamed `omniroute` → **`Omniroute:20128`** and the key moved from an inline `api_key` to a
|
||
**`key_env`** indirection. The snippet now resolves the key from `key_env`. A **new step 3b** documents the
|
||
**`omni-101` quirk** — its `/v1/models` returns `Invalid API key` even with the correct key while
|
||
`/v1/chat/completions` **works** (a `smart-route` call returned `minimaxai/minimax-m3.1-flash`), so a models
|
||
check on that node is a **false failure**. Also corrected the typo "config slocated" → "located". Re-tested 2026-10-26: LiteLLM 401 unauthed (expected), CT600 OmniRoute 6,480 models, WebUI `/health` 200,
|
||
opencode/nous/ollama fallbacks all 200. - **`runbooks/update-hermes-safely.md`** — added the **managed-runtime dependency re-assert step**
|
||
(`hermes-runtime-deps-guard.sh`) after an update, since the updater can leave a runtime without
|
||
`fastembed`/`dotenv`/`mnemosyne`. Clarified `--branch main` vs the **stable** channel (currently **v0.22.5**,
|
||
local **v0.22.3**). Version and `last_tested` brought to 2026-10-26. - **`incidents/2026-10-03-ct450-hermes-update.md`** — added a **Current state** section: both instances now run
|
||
**v0.22.3** (`5b147ab036d6`), CT450 is **0 behind `origin/main`**, and the CT450 Mattermost timeout is still an
|
||
infrastructure condition (CT255 `stopped`), not an update regression. - **`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`** — added a **Current state** section: the
|
||
follow-up is **satisfied** — `fastembed` 0.8.1 imports in **both** managed runtimes (`19da945341ad4794ac2bc8ff9fdd3643`,
|
||
`d41b16c8b6674454a4a819bd4acced1f`) and 0.8.0 in the legacy venv, `FASTEMBED_CACHE_PATH` pinned. Current
|
||
coverage figures recorded; both tiers above the 95 % threshold, so monitor job `b63ed5817212`
|
||
(`last_status: ok`) has nothing to fire on. - **All 9 corrected pages verified at `http://127.0.0.1:9120/<path>` — every one returns 200**, and the served
|
||
`infrastructure/tailscale` contains `octoagent`, `omarchymac-1` and `54 non-funnel`. - **No secrets written.** Job IDs, incident IDs, base URLs, key *env var names*, counts, paths and timestamps only. ### Vault mirror drift (2026-10-26, measured not eyeballed)
|
||
|
||
The vault at `/home/hermes/wiki` tracks **125 files** under **two** conventions:
|
||
`agent-estate-wiki/<path>` (28 files — the Obsidian-visible tree the recent audit
|
||
commits write) and `agent-estate-wiki/wiki/<path>` (93 files — the fuller mirror). This run forward-synced **every page it touched** into both conventions where the
|
||
directory existed — **22 copies, all sha256-verified against the authoring source**
|
||
before the vault commit. Two vault copies differed from the authoring tree
|
||
(`infrastructure/tailscale.md`, `systems/skills-index.md` under `wiki/`); both were
|
||
confirmed to be **older versions** (2026-10-21 vs authoring HEAD 2026-10-25), so
|
||
no direct vault edits were at risk. **Remaining drift (left for an explicit request — a full mirror pass is bigger than an
|
||
audit's mandate):**
|
||
|
||
| Convention | match | stale | missing |
|
||
|---|---|---|---|
|
||
| `agent-estate-wiki/<path>` (top-level) | 25 | 2 | 80 |
|
||
| `agent-estate-wiki/wiki/<path>` | 34 | 58 | 15 |
|
||
|
||
The two top-level **stale** files are `current-state.md` and
|
||
`reports/communities-wa-social-media-audit-2026.md` — both dated historical records
|
||
rather than drift. The vault repo is **local-only** (no remotes); it was committed,
|
||
not pushed. ## [2026-10-27] audit | Nightly wiki audit — window 2026-10-26 00:00 → 2026-10-27 00:5x AWST: a planned restart orphaned a live cron run
|
||
|
||
- **A *planned* gateway restart orphaned a cron run — the second instance of the 2026-10-20 orphan class, and
|
||
this time it was loud.** `gateway.log`: `Received SIGTERM as a planned gateway stop` at **14:33:35**
|
||
(`under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15`). The drain then logged
|
||
`Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work`,
|
||
and at 14:34:29 `Skipping .clean_shutdown marker — drain timed out with interrupted agents`. The victim was
|
||
**`ea79a2107e36` (Mnemosyne Health Monitor)**, claimed **12:01:13** — **2 h 34 m** in. It closed **14:35:59
|
||
`failed`** on a string this wiki had not seen: **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its
|
||
durable fire claim ownership`**. Unlike the 10-20 orphan it raised **two `cron_incidents` rows** (14:33:37
|
||
`ea79a2_c3ab2d784ba3` *"Cron execution unverified; reconcile `admissions/724a69108b…json`"*; 14:35:59
|
||
`ea79a2_a4661554a3c2`, the RuntimeError) and **delivered** the failure. → new
|
||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
|
||
- **The Mnemosyne Watchdog did its job.** `96eb95ce12f2` red-flagged twice (14:46:12, 15:01:15,
|
||
`Script exited with code 1`) reacting to the same log `ERROR`, then **self-resolved at 15:16:12** when the
|
||
line aged out of its 15-minute window. Both rows (`96eb95_0c89bd373a0a`, `96eb95_e32174134546`) are closed. - **Window tally and fleet:** 2026-10-26 settled **169 `completed` / 3 `failed`**, and **all three failures are
|
||
this one interruption** (Health Monitor + the two watchdog reds); 2026-10-25 settled **172 / 2**. Fleet
|
||
unchanged: **28 jobs, 25 enabled / 3 disabled**. **Non-clean `last_status` 5 → 4** (`c0a9fa314542` and
|
||
`08eb86f02d37` cleared to `ok`; `ea79a2107e36` joins as `error`, streak 2). **Unclosed incident rows 2 → 3**. **Pins unchanged.** One gateway start (14:35:18, teardown 54.06 s). **17 sessions in window** (15 `cron`,
|
||
1 `ios`, 1 `telegram`). - **One incident row closed — the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83` **auto-closed
|
||
09:13:00** when the radar's 09:00 slot ran clean; no execution row carries the resource-pressure string on
|
||
10-25 or 10-26. Status stays `monitoring` (the single-node exposure is unchanged). Updated
|
||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`. - **⚠ Two quiet degradations: a completed run that delivered nothing useful, and a broken delivery thread.**
|
||
(a) The radar's 07:00 slot closed **`completed` at 09:10**, but its delivered text read **“RUN INCOMPLETE —
|
||
tool-call budget exhausted”** — no Notion writes, no cover letters — so the day's backlog was executed by hand. (b) `fad5a86979d4`'s 03:08 slot completed but ended `delivery_failed`: **`Thread 34795 not found`** — the same
|
||
dead thread the gateway's own 14:33:37 shutdown notice hit. → `runbooks/job-radar.md`
|
||
- **The radar's cover-letter pipeline changed the same morning.** The PDF is now printed from the **canonical
|
||
rendered HTML** via Chromium (`build_cover_letter_pdf_from_html.py`, created 10:37) instead of ReportLab, so a
|
||
download matches the on-screen letter; the generation default moved to **`deepseek-paid-smart`** (10:47) and
|
||
`job_dashboard_server.py`'s regenerate timeout went **120 s → 600 s** (10:48, server restarted). → `runbooks/job-radar.md`
|
||
- **Files:** new `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`; updated
|
||
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md`,
|
||
`incidents/2026-10-16-omniroute-503-resource-pressure.md`, `runbooks/job-radar.md`,
|
||
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.**
|
||
No secrets written. - **Unresolved:** why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget); whether `failure_streak` should count the *unverified* row (one interruption currently reads streak 2); and the
|
||
**`Thread 34795`** delivery target needs re-pointing, or the Monday 03:08 digest will keep falling back to the
|
||
unthreaded chat. ## [2026-10-28] audit | Nightly wiki audit — window 2026-10-27 00:00 → 2026-10-28 00:2x AWST: the radar's tool-call budget ran out a second time — and this time the run is recorded as failed
|
||
|
||
- **Window 2026-10-27: 172 `completed` / 1 `failed` (+ this audit `running`).** The failure is the radar's
|
||
budget overrun, and it is the story of the night → new page
|
||
[[incidents/2026-10-27-job-radar-budget-exhausted]]. - **The radar's tool-call budget was exhausted for the second consecutive day — and this time the run is
|
||
recorded as `failed`.** `dca8482e4f76` ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed` and
|
||
delivered *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"*: discovery, verification,
|
||
dedup and classification completed; **Notion ingestion, cover-letter generation and the final cover-letter
|
||
audit never ran**. Notion DB `0f90ba2b-8b10-4d02-a62c-6f692d5b1168` held **451 entries** with **0** added
|
||
for 2026-10-27 (Agent — Rhino 402 / Chaos 12 / Manual 2 / blank 35; Source — SEEK 295 / LinkedIn 88). Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** at this audit
|
||
(`failure_type: auth` — which does not describe a budget fault). The previous day's identical overrun closed
|
||
**`completed`** and was found only by reading the delivered text — that is why the 10-27 row matters. - **Response, same day: the radar is now two jobs.** Stage 1 `dca8482e4f76` **Daily Marketing Job Radar -
|
||
Discovery** (`0 7 * * 1-5`, `deliver: local`, failures → Telegram topic 1030) does
|
||
`DISCOVERY → VERIFY → DEDUP → SCORE`, writes a handoff file and triggers stage 2. New stage 2
|
||
**`5fc292ff0f3b`** **Daily Marketing Job Radar - Write and Letters** (`30 11,14 * * 1-5`, Telegram topic
|
||
1030, created **2026-10-27T17:34:15**) lands the Notion entries, cover letters, missing-letter audit and
|
||
report. Two stage-2 test fires on 10-27 (**17:35:00 → 17:36:52**, **17:38:38 → 17:43:42**) both
|
||
`completed`, delivery `suppressed`, `last_status: ok`, streak 0. Pins unchanged (`custom:omni-101` /
|
||
`smart-route`). **Fleet 28 → 29 jobs (26 enabled / 3 disabled).** → `runbooks/job-radar.md`
|
||
- **The 2026-10-26 orphan class did not recur; its two incident rows closed.** `ea79a2_a4661554a3c2` and
|
||
`ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** — unclosed rows fleet-wide **3 → 2** (the
|
||
2: `0e753e_e7f8194a7d06` Facebook birthday monthly scrape, open since 2026-10-01 09:00:55, and the new
|
||
`dca848_e93a538ade6b`). No 10-27 execution row carries the orphan strings and the watchdog alert log has no
|
||
10-27 entry. → `incidents/2026-10-26-planned-restart-orphaned-cron-run.md`
|
||
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
|
||
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Also non-clean: `0e753e9720fd` (error),
|
||
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). - **Two config corrections from live probes.** `tts.provider` is **`openai`** pointed at the **Mac mini speech
|
||
server** (`https://anthonys-mac-mini.kangaroo-eel.ts.net/v1/speech`, model `phonon`, voice `alloy`) — the
|
||
page still claimed ElevenLabs from 2026-10-10 (change date not established, so the old bullet is marked
|
||
superseded rather than deleted). Live voice uses `voice.speech_server.url` =
|
||
`wss://anthonys-mac-mini.kangaroo-eel.ts.net:8443/v1/`; `GET /v1/pool` → `{"size":2,"in_use":0}`, both
|
||
units `idle`, while `/v1/models` returns `{"detail":"Not Found"}`. `voice.auto_tts` is still `false`, and
|
||
`wake_word.enabled` is `true`. → `systems/hermes-agent.md`, `systems/conduit.md`
|
||
- **`systems/skills-index.md` was regenerated by the 2026-10-27 03:00 slot but never committed** — the
|
||
authoring tree was dirty with `updated: 2026-10-27` and a **302 skills / 69 categories** total (against
|
||
index.md's stale **306 / 70 of 2026-10-23**). Freshness at that run: **0 stale / 104 fresh / 4 no-date of
|
||
108 pages** (`freshness-20261027.txt`). Committed, and `index.md` corrected to 302. - **`infrastructure/tailscale.md` peer table re-verified** against `tailscale status --json`; the node-count
|
||
header and any state changes are recorded in that page. - **Files:** new `incidents/2026-10-27-job-radar-budget-exhausted.md`; updated
|
||
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md`, `runbooks/job-radar.md`,
|
||
`systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `systems/conduit.md`,
|
||
`infrastructure/tailscale.md`, `systems/skills-index.md`, `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived.** No secrets written. - **Unresolved:** whether the two-stage split actually fixes the budget problem — the first real 07:00 → 11:30
|
||
cycle (stage 1 handoff → stage 2 writes) had not run at audit time; why a budget exhaustion is classified
|
||
`failure_type: auth`; and the 10-27 radar incident row will only close when a clean stage-1 slot runs. ## [2026-10-29] audit | Nightly wiki audit — window 2026-10-28 00:00 → 2026-10-29 00:3x AWST: the radar's two-stage split ran for real and landed nothing, and the stocktake died on a 400 that never alerted
|
||
|
||
- **New page:** [[incidents/2026-10-28-job-radar-stage1-iteration-cap]] — stage 1 (`dca8482e4f76`) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s), closed `failed`, `failure_streak: 2`, on *"hit the agent iteration cap before the handoff could be written"*. No handoff file existed (`~/.hermes/radar-ref/handoff/` held only `2026-10-27.json`), so **both stage-2 slots (`5fc292ff0f3b`, 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43) `completed` with `delivery_outcome: suppressed`** — the no-op recovery rule proven in production. Second consecutive stage-1 failure at a *different* ceiling (tool-call budget on 10-27 vs iteration cap on 10-28). - **Updated:** [[incidents/2026-10-27-job-radar-budget-exhausted]] (follow-up section answering its open questions: handoff never written, stage 2 no-oped correctly, its incident row did **not** close, unclosed rows 2 → 4) · [[runbooks/job-radar]] (10-28 slot write-up; "first real cycle" marked as run-and-empty) · [[systems/scheduled-tasks]] (new 2026-10-29 window bullet; prior bullet demoted to *Prior window*; the two in-window job rows refreshed) · [[systems/browser-backend]] (dated correction: Camofox restored, `browser-steel` plugin removed, endpoint dark, `STEEL_BASE_URL` residue in 10 live processes) · [[comparisons/steel-vs-camoufox]] · [[decisions/rollback-steel-to-camofox]] (2026-10-29 update) · [[systems/skills-index]] (stocktake-run note) · `current-state.md` (new 2026-10-28 section) · [[index]]. - **Stocktake folded, not paged:** `1a8e90ea2271` ran 03:00:28 → 03:37:58, closed `failed` (streak 1, delivery `suppressed`) on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` after regenerating `systems/skills-index.md` (302 skills) and `raw/stocktake/freshness-20261028.txt` (**2 stale / 103 fresh / 4 no-date of 109 pages**); it left both uncommitted and filed incident `1a8e90_f57ab4cafab8` in state **`detected`** with no alert. The audit committed the two artefacts, clearing the dirty tree. - **Live probes (2026-10-29 00:3x AWST):** `jobs.json` 29 jobs / 26 enabled / 3 disabled, 5 non-clean `last_status`, pins 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`; `executions.db` — 2026-10-28 settled **168 / 2 / 1 running**, 2026-10-27 now **173 / 1**, 2026-10-29 so far 6 / 1 running; Steel: ping 2.5 ms but CDP `:9223` and REST `:3000` refuse, tunnel `/steel` **502**, `steel.martinwa.org` unresolved; no Hindsight service on this box (`:8888` closed, no `hindsight` units) so [[systems/memory-backend]] and [[comparisons/memory-backends]] were left unchanged. - **Removed/archived:** none. **Secrets written:** none. - **Unresolved:** the radar's discovery stage still cannot reach its own handoff write (two ceilings in two days); `1a8e90_f57ab4cafab8` sits `detected` without an alert; both radar incident rows close only on a clean 07:00 weekday slot. ## [2026-10-29] update | Steel Browser fully removed from the estate — Camofox is the sole backend
|
||
|
||
- **Root cause of the long-standing `cloud_provider: steel` residue:** the 2026-07-30 rollback disabled the `browser-steel` plugin and set `browser.engine: camofox`, but left `browser.cloud_provider: steel` in `~/.hermes/config.yaml`. Every browser tool call then routed through `_instantiate_explicit_cloud_provider()` and failed with *"no registered browser plugin has that name: steel"*. Because `engine` was already correct, the split config was inert-but-wrong and survived ~3 months of audits. - **Hermes (CT460):** `browser:` block now reads exactly `cloud_provider: camofox`, `camera_provider: camofox`, `engine: auto`, `cdp_url: ""`. The `browser-steel` plugin directory was **deleted**, and the `browser-steel` key was removed from both `plugins.disabled` and `plugins.entries`. `STEEL_BASE_URL` was dropped from `~/.hermes/.env`. **Hermes core source contains zero Steel references**, so there is no Steel fallback path in the tree — returning to Steel now means redeploying it. - **CT201 (runtipi):** the entire Steel deployment was torn down — containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network` Docker network, the images `steel-browser-api:fixed` and both ghcr.io Steel images, and the project dir `/root/steel-browser` (~6 GB reclaimed; disk 70G → 64G used). The Tailscale Serve routes `/steel` (5173) and `/steel-api` (3000) were deleted with `tailscale serve --set-path <path> off`; CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`. Nothing listens on 3000, 5173 or 9223. - **CT333 (clawtest):** a **second, separate and orphaned** Steel install was found and removed — the `steel-browser` container (`ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30, publishing port 3000) and its image. It had no Serve route, no entry in `~/.zeroclaw/config.toml` (which uses `backend = "agent_browser"` on 127.0.0.1:9515), no reference from Nanobot's config or the CLI-anything browser harness; its last CDP navigation was 2026-10-20, it received no requests after 2026-10-21, and its page crashed 2026-10-22. - **Estate-wide sweep:** every LXC on the PVE host was checked — no Steel containers, images, networks or listeners remain anywhere. Homarr, the homepage dashboard and the runtipi app configs had no Steel references. - **Gateway restarted 01:21 AWST** to clear `STEEL_BASE_URL` from the inherited process environment; the new gateway tree (PID `3913841`) reads `STEEL_BASE_URL=0`, and Telegram, Discord, ntfy, email and Home Assistant all reconnected cleanly. The dashboard unit and `agentmail-webhook.service` still carry the variable in their inherited environment, but **no core code reads it**. - **Reversibility preserved:** CT201's compose file and Dockerfile are archived under `/root/_archived-steel/`, plus `/root/_archived-steel/steel-cache-archive-20261029.tgz`; the images remain re-pullable from ghcr.io. Backups: `config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z`, `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z`, `.env.bak-pre-steel-cleanup-20261028T220947Z`. - **Camofox verified healthy** after every change: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**, `/tabs` → `{"running":true,"tabs":[]}`, and live navigation confirmed working. - **Pages corrected (15):** [[systems/browser-backend]] · [[decisions/rollback-steel-to-camofox]] · [[decisions/index]] · [[comparisons/steel-vs-camoufox]] · [[comparisons/camoufox-vs-chrome]] · [[infrastructure/docker-services]] · [[infrastructure/tailscale]] · [[infrastructure/hosts]] · [[systems/homepage-dashboard]] · [[concepts/browser-automation]] · [[runbooks/recover-docker-service]] · [[current-state]] · [[index]] · [[agents/nanobot]] · [[systems/scheduled-tasks]]. - **Corrects the 2026-10-29 00:3x audit entry**, which described Steel as merely "not reachable / endpoint dark". It is **removed**, not down. Per SCHEMA rule 9 the superseded claims are marked, not deleted. - **Removed/archived:** none. **Secrets written:** none (variable names only; no key or URL values). - **Unresolved:** the obsolete CT333 line in Nanobot's Obsidian vault (`Areas/Homelab/architecture.md`, "Steel Browser | Clawtest | http://localhost:3000") still stands — that vault is outside this wiki and was not modified. Also noted, unrelated to Steel: `Browser Use viewing unavailable: unsupported Hermes browser resolver` has logged in the gateway since 2026-10-11. ## [2026-10-29] stocktake | General Agent Estate Wiki maintenance — Tailscale table re-verified, and a 23-day-old "never re-authenticated" claim turned out to be false
|
||
|
||
- **Script:** `wiki-maintain.sh` ran clean (exit 0). `systems/skills-index.md` regenerated from the live skill tree — **302 skills in 69 categories**, with a new note that **45 skills** are parked under `~/.hermes/skills/.archive/` and excluded from the count. (303 `SKILL.md` files exist outside `.archive/`, but `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` is a nested alias duplicate of `productivity/obsidian-git-sync/SKILL.md`, so **302 unique skills** is the correct figure.) Freshness report written to `raw/stocktake/freshness-20261029.txt`; Quartz rebuilt (110 pages → 644 files) and deployed to `public/`. - **Freshness report:** **2 stale / 105 fresh / 3 no-date** of 110 pages. The 2 stale were `incidents/2026-10-01-cliproxy-connection-auth-expired.md` and `incidents/2026-10-05-dead-model-pin-repin.md`, both at **23d**. - **Updated [[infrastructure/tailscale]]** — State column rebuilt from `tailscale status --json`; **11 rows changed**. Two were **state transitions**: **`sftpgo` (CT299, pve) is back online** (the 2026-10-28 audit had recorded it `offline <1d`; `tailscale ping` → pong in 0s, `systemctl is-active sftpgo` → `active`, `tailscale ip -4` inside the guest → `100.88.223.70`), and **`m93p` is online despite a stale peer record** (`Online: false` in the JSON, but `tailscale ping` → pong in 1ms and SSH over the tailnet returned `m93p / up 11 days`) — the **second** occurrence of that pattern after 2026-10-26. The other 9 rows were offline-age drift of exactly +1 day. Node count unchanged at **54** (53 live IPv4 peers + `hermes`); JSON reports **76 peers — 53 with a `100.` IPv4, 23 IPv6-only funnel ingress**. **No missing peers, no IP mismatches, CT601 still absent from both hypervisors.** Container identity re-confirmed from **inside** all 34 guests via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; the 3 stopped guests (CT209/CT255/CT555) and the 3 running-but-unconnected guests (CT105/CT1264/CT100) confirmed. - **Updated [[incidents/2026-10-01-cliproxy-connection-auth-expired]]** — the residual claim *"the connection credential itself was never re-authenticated"* is **falsified**. A live `POST /v1/chat/completions` for `cliprox/terra` against `custom:Omniroute:20128` (CT600) returns **HTTP 200** — the connection has since been re-authenticated, independently of the incident's remediation. The same pin on `custom:omni-101` (CT101) still returns **HTTP 401** (`No active credentials for provider: cliprox.`), so the two nodes carry separate credentials. Claim marked superseded rather than deleted (SCHEMA rule 9); the remediation itself (moving 3 jobs off the pin) was never reversed and remains correct. - **Updated [[incidents/2026-10-05-dead-model-pin-repin]]** — every claim re-probed live: all 7 jobs are still on `custom:omni-101` with routing aliases (`free-stack` ×6, `smart-route` ×1) ✔; both aliases return **HTTP 200** ✔; `ollama-cloud/qwen3.5:397b` still **HTTP 410** (retired 2026-09-25) ✔; `nvidia/deepseek-v4-flash` still **HTTP 400** (absent from the live catalog) ✔; `command-code/gpt-5.6-terra` now returns **HTTP 401** (credential) rather than the original **403** (plan-gated) — the pin is still dead, only the failure signature changed. Frontmatter `updated`/`verified_on` bumped to 2026-10-29 with `created` left at 2026-10-06 as the data period. - **Updated [[systems/scheduled-tasks]]** — the Known-Issues entry repeating the "never re-authenticated" claim now carries the correction. The page's fleet figures were re-checked against live `jobs.json` and remain accurate: **29 jobs / 26 enabled / 3 disabled**, 5 non-clean `last_status`. - **Verified, no change needed:** [[systems/homepage-dashboard]] (CT201 `ss -ltn` shows 8084 and 8085 listening, **no 8082** listener, Homarr on 8102 — all as the page claims); CT201 Serve routes (`/` → 8085, `/mgr` → 8084 on **both** `:443` and `:8443`, exactly as documented, Steel routes still gone); CT460 Serve routes (12 paths on `:443` + `:8443/webhook` Funnel + `:9443`, matching the page). - **Not changed, flagged for a human:** the vault mirror at `/home/hermes/wiki/agent-estate-wiki/` has drifted on **both** conventions — top-level **35 match / 3 stale / 72 missing**, `wiki/` **41 match / 53 stale / 16 missing**. A full mirror pass is larger than a stocktake's mandate, so it is recorded here rather than performed. - **Removed/archived:** none. **Secrets written:** none (credential names and HTTP status codes only — no key or token values). - **Unresolved:** `cliprox/terra` remains **401 on CT101** — the CT101-side connection still needs a dashboard reconnect if anything is ever to target that pin from that node. ## [2026-10-30] audit | Nightly wiki audit — window 2026-10-29 00:00 → 2026-10-30 00:0x AWST: the radar's split finished end to end for the first time, and Conduit voice calls work after three faults in series were fixed
|
||
|
||
- **New page — [[incidents/2026-10-29-conduit-voice-silence]]:** every Conduit voice call connected and then died
|
||
silently. Three faults sat in series, each one masking the next: (A) `voice.speech_server.url` was the truncated
|
||
`wss://…:8443/v1/`, which the plugin dials **verbatim**, so the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the **12 h default**, so an expired phone session got **WS 4401**
|
||
with no Hermes turn and no error on the device (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent
|
||
call); (C) **both** host providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq`
|
||
→ **403/Cloudflare 1010** (blocked, not mis-keyed). Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL →
|
||
**2 592 000 s (30 d)**, `stt.provider: local` (faster-whisper `base`), `tts.provider: ollama_local`; dashboard
|
||
restarted **08:03:26**. Verified by driving the whole call path for real, then by **two live calls at 08:38:04 and
|
||
08:49:36** — the first entries ever in `conduit-calls.json → history`, so the memory fix is no longer evidence-free. - **Updated [[systems/conduit]]** — new runtime-checks section, the two real calls on record, the incident cross-reference,
|
||
and `verified_on: 2026-10-30`. - **Updated [[systems/hermes-agent]]** — new dated section **2026-10-30 00:0x AWST** for the voice path (fault table +
|
||
why it stayed invisible + the two-TTL-keys warning + backups), and the `tts.provider` block now carries a second
|
||
*superseded* note recording the live values (`ollama_local` / `local`, `/v1/realtime`). - **Updated [[systems/scheduled-tasks]]** — new Count bullet for the 2026-10-30 window, the previous bullet demoted to
|
||
*superseded*, stage 1's delivery column corrected to `local` (verified live: stage 1 no longer delivers), both radar
|
||
rows annotated with their 2026-10-29 result, and a new **Resolved 2026-10-29** block under Known Issues. - **Updated [[runbooks/job-radar]]** — new dated slot section **2026-10-29 07:00** (handoff written → consumed →
|
||
delivered: 170 discovered / 33 duplicates avoided / 37 verified; **20 pages + 2 cover letters**,
|
||
`missing_cover_letter_count 0`, Notion board **455 → 475**), plus the note that the second real cycle completed. - **Updated [[incidents/2026-10-27-job-radar-budget-exhausted]]** and
|
||
**[[incidents/2026-10-28-job-radar-stage1-iteration-cap]]** — both **resolved**. Stage 1 ran 07:00:59 → 10:16:12
|
||
(3 h 15 m 13 s), closed `completed`, and both ledger rows it had opened (`dca848_e93a538ade6b`,
|
||
`dca848_0a2f03b5952b`) closed `resolved` at 10:16:12. Stage 2 consumed the handoff at 10:40:59, closed `completed`
|
||
at 10:44:48 and delivered the same second; the 11:30 and 14:30 slots no-oped as designed. - **Updated [[current-state]]**, **[[index]]** — new Recent Changes block and the refreshed page descriptors. - **Cron health:** window **2026-10-29 settled 176 `completed` / 0 `failed`** — the first zero-failure day since
|
||
2026-10-24 — with **10 deliveries, all 10 `delivered`**. Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status` **5 → 3** (all three are stale re-runs of long-standing rows) and every `failure_streak`
|
||
reads **0**. **Unclosed incident rows 4 → 1**, the lone survivor being the Facebook monthly's
|
||
`0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Pins unchanged. - **Vault mirror catch-up:** the drift flagged by the 2026-10-29 stocktake is partly cleared — the pages changed by this
|
||
audit were mirrored into `/home/hermes/wiki/agent-estate-wiki/`, including `systems/conduit.md`,
|
||
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `systems/hermes-agent.md`, `current-state.md`, `index.md` and
|
||
`log.md`. The wiki-maintained pages under `infrastructure/` remain a separate, known gap. - **Removed/archived:** none. **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a
|
||
key or token value. ## 2026-10-30 — Stocktake (03:00 AWST): 0 stale by age, but 5 pages had drifted from live state
|
||
|
||
- **Ran `wiki-maintain.sh`.** Regenerated `systems/skills-index.md` (**302 skills / 69 categories** — unchanged; the
|
||
generator de-dups one nested duplicate, so 302 is the correct unique-skill figure), wrote
|
||
`raw/stocktake/freshness-20261030.txt` (**0 stale / 109 fresh / 3 no-date of 112 pages**), rebuilt Quartz
|
||
(654 files) and deployed to `public/`. - **Freshness is not accuracy.** The age scan found nothing, but a live-vs-page diff against the running estate found
|
||
real drift on five pages. All corrections below were verified against live sources before being written. - **[[infrastructure/tailscale]] — node table rebuilt.** `tailscale status --json` now reports **50 non-funnel nodes**
|
||
(49 live IPv4 peers + `hermes`), not the 54 the page claimed. **2 IP changes** (`anthonys-mac-mini` 100.74.17.15 →
|
||
100.76.60.37; `omarchymac` 100.84.180.69 → 100.126.173.115 — the peer formerly listed as `omarchymac-1`) and **4 nodes
|
||
dropped out of the tailnet entirely** (`anthonys-imac`, `cachyos`, `omarchymac-1`, `omni`). Every offline age refreshed. `m93p` reads `Online: false` for the **third** consecutive audit yet is plainly up (`tailscale ping` → pong, SSH → `up 12
|
||
days`), so the row records it online. Total peers **72** (49 IPv4 + 23 IPv6-only funnel). CT460's `:443` route set
|
||
matched the page exactly; CT201 serves only `/` → 8085 and `/mgr` → 8084 on both `:443` and `:8443`. - **[[systems/scheduled-tasks]] — fleet ledger corrected.** Live `jobs.json` shows **29 jobs (25 enabled / 4 disabled)**, not
|
||
the 28/25/3 on the page: `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused 2026-10-30 01:04:22** after its
|
||
00:01:08 slot closed **`unknown`** on a gateway-connection failure. Non-clean `last_status` is **4** (not 3), and the
|
||
standing "every `failure_streak` reads 0" claim was false (`ea79a2107e36` = 1). "Failing (delivery only)" is **2**, not 1
|
||
(`fad5a86979d4` joined `066bb5191280`). The retired DHL row was dropped from the job table (it lives in Removed Jobs). Seven job rows carried **stale `error` claims** for jobs that have since cleared to `ok` — each was marked superseded
|
||
with a dated correction rather than rewritten (SCHEMA rule 9). The `executions.db` retention floor note was corrected to
|
||
the live **2026-10-24 09:22:36**. Window tallies: **2026-10-29 = 177 / 0** and **2026-10-28 = 169 / 2** (both +1 on the
|
||
page — the 23:59 Wiki Audit row had still been `running` at audit time). - **New incident page: [[incidents/2026-10-30-mnemosyne-health-monitor-paused]].** The Mnemosyne Health Monitor's own
|
||
failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) —
|
||
`deliveries.db` holds **zero rows for 2026-10-30**, so the incident row is the only trace. The job was **not re-enabled**
|
||
and the gateway was **not restarted** by this run; root cause is **unverified** and recorded as such. - **[[infrastructure/hosts]] — four corrections.** CT201's LAN address is `192.168.178.77`, not `192.168.178.166`; CT450's Hermes Agent is **v0.22.5** (2026.9.24),
|
||
local **`bf0203a794a0`** — the same build as CT460 — so the page's "v0.22.3 / 5b147ab036d6" was wrong on both
|
||
(a stale *root-level* `/usr/local/bin/hermes` on CT450 still reports v0.19.0 and is **not** the live agent —
|
||
the live one is the `hermes`-user install under `~/.hermes/hermes-agent`); the cron count is now 29/25/4; and
|
||
the orphaned `omni`
|
||
peer has **expired out of the tailnet entirely** (no peer named `omni`, no peer at 100.93.204.4), so the "offline 69d"
|
||
description no longer applied. The **skills count of 302 was left alone** — it is correct. - **[[systems/homepage-dashboard]] — five corrections, and a second pass to fix my own first pass.**
|
||
"All Apps" is **114 → 123 live apps**, not 122 (the API dedupes by `(name, normalised href)` across groups; the literal `All Apps` group in `services.yaml` holds 122 and is not what the tab bar counts). The grid is a
|
||
responsive `repeat(auto-fill, minmax(230px, 1fr))` (served `/dash.css` line 3640), not a fixed 5-column grid. The tab list was wrong — but **my first correction was wrong too** and had to be redone: the live tab bar is
|
||
built from the service *groups* (`renderTabs()` iterates `state.groups`), **not** from the `layout:` block in
|
||
`settings.yaml`. That block and its `tab:`/`columns:` keys are **inert** — `grep -c "Overview" /dash.js` = 0 and
|
||
`grep -c "columns" /dash.js` = 0. There is **no "Overview" tab**; the live tabs are `☆ Pinned`, `All (123)` and
|
||
the **14 groups** (AI Quotas 3, Infrastructure 13, Everyday 8, Agents 20, Media 11, AI & LLM 17, Tools 20,
|
||
Storage 6, More 13, Resources 6, System Health 0, Subscriptions 1, Quick Launch 2, Uncategorised 3). The saved
|
||
theme is **`{"pair":"Boarding Pass","mode":"light"}`** — one of **59** theme pairs; its light-mode amber is
|
||
`#795000`, so the old "dark base with warm amber (#d4a04a)" was wrong on both counts. **`manager.py` runs inside
|
||
the `homepage-manager` container** (its container `Cmd` is the `pip install … && python3 /opt/homepage/manager.py`
|
||
line, confirmed via `docker inspect` and `docker top`); only **`asset_proxy.py`** is a host process
|
||
(`0.0.0.0:8085 users:(("python3",pid=4038))`). And the "no Traefik container exists" evidence was wrong — the
|
||
`runtipi-reverse-proxy` Traefik container exists but is **stopped** (`Exited (255) 2 months ago`); the *conclusion*
|
||
that Traefik does not occupy 443 stands. The same evidence error was corrected in [[infrastructure/tailscale]]. - **[[current-state]]** and **[[index]]** updated with the new incident and the corrected fleet numbers. - **Verified, not assumed:** CT600 **running** and CT601 **absent from both hypervisors** (`pct config 601` → no such
|
||
config on `pve` or `m93p`); the CT201 `:443`/`:8443` Serve tables; CT201 listeners 5173/3000/8082 (**none** — Steel's
|
||
removal confirmed); the homepage service groups and theme; and every cron counter, tally, incident row and delivery row
|
||
(read from `jobs.json`, `executions.db` — whose `cron_incidents` table lives inside it, the standalone
|
||
`cron_incidents.db`/`incidents.db` files are empty stubs — and `deliveries.db`). - **Not changed:** `systems/skills-index.md` count (302 is correct), CT460's Serve route table (matched live exactly), and
|
||
the steel-removal records across the wiki (confirmed still true). - **Secrets written:** none — provider names, key *names* and HTTP status codes only, never a key or token value. ## [2026-10-31] audit | Nightly wiki audit — window 2026-10-30 00:00 → 2026-10-31 00:3x AWST
|
||
|
||
**Verdict: one large undocumented event found and written up, and one *documented* incident found to have been
|
||
recorded wrong.**
|
||
|
||
- **NEW PAGE — [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].** CT460 was dark for **5 h 30 m** on
|
||
2026-10-30 and **no monitor alerted**. Evidence: **zero `executions.db` rows between 10:58:20 and 16:37:21**; PVE host `pve` boot time **2026-10-30 16:24:00**; CT460 boot **16:28:00**; the previous gateway life
|
||
(`pid=3913841`, started 2026-10-28T17:21:58Z) logged **UNCLEANLY at 16:35:50** (*"no exit path ran —
|
||
SIGKILL / OOM / VM death"*); scheduler resumed **16:37:21** with an **8-run catch-up burst** through 16:38:22. The day still closed **138 executions / 0 failed** against 10-29's **177 / 0** — **≈39 lost slots, no failures**. Root cause **not established**: no OOM evidence locally or on the PVE host, so the ledger string is read as a
|
||
candidate list, not a diagnosis. **Still open:** the two jobs that could have detected it (`cbfe434a3b5e` NFS
|
||
mount monitor, `1a8e90ea2271` m93p LXC boot monitor) are disabled. - **CORRECTED — [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] was right about the record and wrong
|
||
about the run.** `ea79a2107e36`'s 2026-10-30 00:01:08 slot was **still running** at the 03:3x reading; it closed
|
||
**`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**, auto-closing incident
|
||
`ea79a2_d93c3acdc4f2` **`resolved` at 2026-10-30T10:02:48.662+08:00**. Live job state is now `enabled: true`,
|
||
`last_status: ok`, `failure_streak: 0`. The page's *"zero `deliveries.db` rows / the notice was never
|
||
delivered"* claim was an artefact of reading the day at 03:3x: 2026-10-30 finished with **8 delivery rows, all
|
||
`delivered`** (latest 18:46:04). Page frontmatter moved `active` → **`resolved`**, with the old snapshot kept
|
||
below as explicitly historical. ⚠ Its **12:01 slot was lost to the host outage** and is the still-`running`
|
||
16:37:21 catch-up. **Reconciliation lesson: a slot mid-flight is not a failure — check `finished_at`, not
|
||
`last_status`, before writing an incident.**
|
||
- **Reconciled — `systems/scheduled-tasks.md`.** Fleet **29 jobs (26 enabled / 3 disabled)** — the "4th disabled
|
||
job" reading is superseded (frontmatter date, fleet block, Total jobs, non-clean `last_status` **4 → 3**,
|
||
failing **1 → 0**, the health-monitor table row, the disabled table row, and a new **"Resolved 2026-10-30"**
|
||
subsection). Non-clean now **3 = 1 `error` + 2 `delivery_failed`**; unclosed incident rows **2 → 1** (survivor:
|
||
the Facebook monthly's `0e753e_e7f8194a7d06`, due 11-01). - **Reconciled** — `current-state.md` (new leading cron entry + superseded marker), `infrastructure/hosts.md`
|
||
(cron row), `index.md` (frontmatter, banner, scheduled-tasks entry, both 10-30 incident entries),
|
||
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`. - **Checked, unchanged:** 12 sessions active in-window (longest `local-40472e0b…`, 1,351 messages), no orphaned
|
||
`running` rows, no incidents opened or closed after 10-30, no obsolete pages to archive, no secrets written. ## [2026-11-01] audit | Nightly review — Conduit 0.122.2, the Local Live fault, a force-drained reload, dead Asana paths
|
||
- **NEW PAGE — [[incidents/2026-10-31-conduit-voice-local-mode-not-applying]].** The app's **"local" voice mode
|
||
never applies** even though its connection test succeeds — because the test checks **socket reachability only**. Every local turn is still impossible: the Mac mini speech server's **LLM proxy is disabled** (LLM endpoint →
|
||
**HTTP 501**, *"start the server with `--enable_llm_proxy`"*), `/v1/usage` reads **`llm_proxy.requests: 0`**
|
||
across 25 observed turns, and its realtime socket **accepts the connection but rejects `session.update` as an
|
||
unknown event**. `voice.mode` is a **phone-side** selector: **no `mode` key exists** in `config.yaml → voice`
|
||
or `conduit-settings.json` (whose only device state is `devices/<UUID>/mode: small` ×3), so "the change did not
|
||
apply" cannot be diagnosed on host config. Evidence from session `20261031_193709_40b9c2` (started **19:37:09**,
|
||
30 msgs / 17 tool calls, still open at the 00:2x reading); `voice.local_live.front_model` was repointed to
|
||
`mistral/voxtral-small-latest` mid-session (`config.yaml` mtime **21:02:30**). **Nothing was changed on the
|
||
Mac.** ⚠ One misread was caught in-session and is recorded as a lesson: `local_live.py:882` is the **only**
|
||
writer of `_local_live_last_request`, inside the chat-completions route — so a "recent `last_request_at`" was
|
||
the investigation's **own probe**, not phone traffic (`last_request_at` is `null`). - **NEW PAGE — [[incidents/2026-10-31-gateway-reload-force-drain-orphaned-run]].** The **18:06:41** gateway reload
|
||
waited the **full 1,800 s** drain cap and force-drained at **18:37:38** (new gateway **18:38:43**), orphaning
|
||
`ea79a2107e36` (**Mnemosyne Health Monitor**, 12:01:06 slot*, **6 h 38 m 09 s** in) which closed **`failed`** at
|
||
**18:39:15** on *"lost its durable fire claim ownership"*. Derivative watchdog reds at 18:39:23 / 18:45:23 /
|
||
19:00:22 cleared **19:15:21**. A **second, clean** restart followed at **19:23:39 → 19:24:40**. Same class as
|
||
the 10-26 planned-restart orphan — **second instance, first on a plain reload**. - **NEW PAGE — [[incidents/2026-10-31-asana-access-failures]].** Both Asana paths on CT460 are non-functional:
|
||
the MCP server `asana` (`https://mcp.asana.com/v2/mcp`, `auth: oauth`, `enabled: true`) has **never
|
||
authenticated** (`~/.hermes/mcp-tokens/` holds only `asana.client.json`, **319 B**, mtime 2026-10-26 01:07; logs read *"MCP OAuth setup failed for 'asana' … no cached tokens found. Run `hermes mcp login asana`
|
||
interactively first"*), and the hosted connector rejects the call shape. The repair needs an **interactive**
|
||
login, which cron cannot perform. - **UPDATED — `systems/conduit.md`** (13,184 → 18,722 B; frontmatter `updated` / `verified_on` → 2026-11-01). Version `0.69.2` / `942ddb8` → **`0.122.2` / `9eac857`** (10-26 reading kept explicitly superseded); the
|
||
⚠ *"uncommitted patch, five releases behind"* warning is **RESOLVED** — the tree is clean and
|
||
**`MEMORY_TIMEOUT_S = 10.0` ships upstream** in `dashboard/api/memory.py` (with a `wait_for` backstop and
|
||
`tests/test_memory.py`), so the 10-23 in-tree edit is no longer needed; dashboard gateway
|
||
`pid 4042003` (10-29 08:03:26) → **`pid 176771` (10-31 19:35:02)**; the **call-store shape change** is recorded
|
||
(`v: 1`; **`history` empty**; **`last_request_at: null`**; `ends` = 2 × `outcome: stopped` at 21:06:05 /
|
||
21:10:48; `settings.quiet_at_screen: true` = #479); data volume 930.0 MB → **938.2 MB** (983,793,664 B) with
|
||
**2,623,879 `events` / 4,384 `buckets` / 504 `windows`** (≈8 MB/day, no retention policy); new dated section; `Open items` — patch item marked resolved, three new local-live items added. - **UPDATED — `systems/scheduled-tasks.md`** (same pass): fleet **29 jobs (26 enabled / 3 disabled)**; 2026-10-31
|
||
closed **160 / 4**; non-clean `last_status` **4** (2 `error` + 2 `delivery_failed`); failing jobs **1**
|
||
(`ea79a2107e36`, `failure_streak` 2); superseded readings marked rather than overwritten. - **UPDATED — `systems/hermes-agent.md`:** Asana MCP auth-state re-verification note (enabled but never
|
||
authenticated) + both 2026-10-31 gateway restarts (18:06:41 force-drain, 19:23:39 → 19:24:40 clean). - **UPDATED — `index.md`:** frontmatter and banner → **2026-11-01**; Conduit entry (version, patch resolved,
|
||
gateway pid, new fault, empty call history); **three new incident entries** added above the 10-30 pair. - **UPDATED — `log.md`:** this entry. - **Checked, unchanged:** ship rows before 10-30 (nothing new); `inc_20261031_000001` (gateway-update) is **still
|
||
open**, `inc_20261030_000001` closed 2026-10-30 18:05; **3** `deliveries.db` rows for 2026-10-31 carry a
|
||
failure error (`child run failed: exit status 1`) — they are the failing runs' own notices, not new faults; no page archived; **no secrets, tokens, passwords or keys written**. - **Probes (context, not findings):** `omniroute-1` `100.78.63.20:20128` → **HTTP 200**; `m93p` `100.104.51.98`
|
||
and `pve` `100.96.100.82` answered neither ICMP, SSH nor `:8006` from CT460 at the 00:2x reading —
|
||
recorded as **unverified reachability from this container**, not as a host outage. - **Mirror catch-up (a whole-tree pass, beyond this audit's own pages).** Because the previous copy bridge only
|
||
ever synced the pages an audit touched, the user-facing vault had silently drifted. This pass re-copied **every
|
||
committed page** in the authoring tree into **both** vault layouts (`/home/hermes/wiki/agent-estate-wiki/` and
|
||
`.../wiki/`) — **61 pages** — which closed the 9-page drift and picked up everything the **2026-10-31** audit had
|
||
changed and never mirrored (`current-state.md`, `incidents/2026-10-30-ct460-host-outage-scheduler-gap.md`,
|
||
`incidents/2026-10-30-mnemosyne-health-monitor-paused.md`, `infrastructure/hosts.md`, `systems/mnemosyne.md`)
|
||
plus 53 older stragglers. **Deliberately left unsynced:** the **6** files the 2026-10-31 03:00 stocktake has left
|
||
**uncommitted** in the authoring repo (`infrastructure/docker-services.md`, `infrastructure/proxmox.md`,
|
||
`infrastructure/tailscale.md`, `systems/messaging-integrations.md`, `systems/omniroute.md`,
|
||
`systems/skills-index.md`) and untracked `raw/stocktake/freshness-20261031.txt` — the mirror convention is
|
||
*"(authoring COMMIT_SHA)"*, so uncommitted content stays out until the stocktake's own commit lands. ## [2026-11-01] stocktake | General Agent Estate Wiki maintenance — Quartz build was broken by a YAML title; the "unfixed memory ranking" incident turns out to have been fixed three weeks ago
|
||
|
||
- **BUILD FAILURE FOUND AND FIXED (this was blocking the whole stocktake).** `wiki-maintain.sh` exited **1** at
|
||
step 3/3 with an esbuild panic. Root cause was **not** esbuild: `incidents/2026-10-31-conduit-voice-local-mode-not-applying.md`
|
||
had an **unquoted YAML `title:`** containing the substring `: ` (from `"voice mode: local"`), so YAML parsed it as a
|
||
mapping entry and Quartz aborted with `bad indentation of a mapping entry (2:27)`. A scan of all **117** pages
|
||
found **exactly one** offending frontmatter line; the title is now double-quoted with escaped inner quotes. **This is a latent trap worth remembering:** any future page title containing `": "` or `":` unquoted will
|
||
silently break the entire Quartz build, and the failure surfaces as an opaque esbuild goroutine dump rather
|
||
than a YAML error. - **STALE PAGE — `incidents/2026-10-09-memory-recall-relevance.md` (23d) — RESOLVED, not stale.** The page's
|
||
central claim was that the recall-ranking fix *"was reported saved and is **not** present in live config"*, and
|
||
that conflict `152` was *"the only open conflict"*. **Both claims are false, and the page was the reason the
|
||
incident looked open for three weeks.** Live `~/.hermes/mnemosyne/config.yaml` reads `vec_weight: 0.5`,
|
||
`fts_weight: 0.3`, `importance_weight: 0.3` (plus `MNEMOSYNE_IMPORTANCE_WEIGHT=0.3` in the environment) —
|
||
**the weights live in Mnemosyne's own config file, not in `~/.hermes/config.yaml`'s `memory.mnemosyne` block
|
||
that the 2026-10-10 probe read.** The 2026-10-10 verification probed the wrong file and concluded "not applied". **The original regression test now passes:** `mnemosyne recall "Are you up to date"` returns only substantive
|
||
current-state facts (scores 0.174 / 0.130 / 0.123) and the stale Uber Eats / Sonos episode
|
||
(`id 32f469aae09d56dc`, still in the DB) **no longer outranks anything**. Conflict `152` was auto-resolved
|
||
**2026-10-29 23:53:47Z** ("low-confidence LLM extraction noise") — it has not been open since. All three open
|
||
items closed; page moved to `status: resolved`. Live tier counts at 2026-11-01: working **941**, episodic
|
||
**1,124**, canonical **90**, conflicts **175** with **11 open** (ids 165–175, all one-batch extraction noise
|
||
sharing a single `fact_a_id`), working embeddings **927/941 (98.5 %)**, episodic **1,124/1,124 (100 %)** — both
|
||
above the 95 % floor, no `reindex` needed. - **UPDATED — `systems/mnemosyne.md`:** stale 2026-10-09 tier-count block marked superseded and a live 2026-11-01
|
||
block added, including the weights' actual home and the resolved-incident pointer. - **UPDATED — `infrastructure/tailscale.md`:** table rebuilt from live `tailscale status --json` — **1 state
|
||
transition and 5 offline-age drifts, no IP changes, no missing peers**. `omarchymac` (100.126.173.115) went
|
||
**offline** (`Online: false`, `LastSeen 2026-10-31 13:20Z`, `tailscale ping` times out — a genuine transition; the JSON `OS` field reads `linux`). `m93p`'s peer record reads `Online: false` **again** — the **4th
|
||
consecutive** stale record — while `tailscale ping` → `pong` and SSH returned `up 1 day, 12:02`, so the row stays
|
||
**online**. Ages refreshed: cloudreve 98→**99d**, freshrss-new 74→**75d**, omarchybook 9→**10d**, openmausbot
|
||
9→**10d**, ubuntu-server-xfce 71→**72d**; `conduit` (55d) and `elementsynapse` (102d) were already correct. - **UPDATED — `infrastructure/docker-services.md`:** **CT201's Tailscale IP was wrong** — the page read
|
||
`100.116.116.116`; live `pct exec 201 -- tailscale ip -4` returns **`100.96.244.39`** (which
|
||
`[[infrastructure/hosts]]` already had right). The CT200 inventory had **three misspelled `overtchat-*`
|
||
entries** (written `overchat-*`) and **one stale entry**, `worldmonitor-seeder-fast`, which is not running —
|
||
the list is now regenerated verbatim from live `docker ps`. **CT333's first container was renamed**:
|
||
`adoring_jemison` → **`thirsty_payne`**. Counts re-verified: CT200 **61**, CT201 **20**, CT333 **8**,
|
||
CT1001 **1** (`sshwifty`), CT460 `ignis` running. - **UPDATED — `infrastructure/proxmox.md`:** storage percentages refreshed against live `pvesm status` on **both**
|
||
hosts (`pve` `fred-qnap` 82.45→**82.77 %**, `SSD1` 63.80→**64.32 %**, `local-lvm` 13.60→**13.75 %**; `m93p`
|
||
`nfs-proxmox-*` 82.45→**82.78 %**, `omb-internal` 58.00→**57.92 %**). `pct list` / `qm list` on both hosts
|
||
re-confirmed — **no CT601 on either hypervisor**, CT255 still `running`, VM 103 `cachyos-m93p` stopped, VM 106
|
||
`openmausbot-vm` running. `m93p` uptime recorded as **up 1 day, 12:02** with a **load average of
|
||
7.66 / 12.80 / 11.32** — elevated, worth watching. - **UPDATED — `systems/omniroute.md` / `systems/messaging-integrations.md`:** re-probed today — both OmniRoute
|
||
nodes answer **401** on `/v1/models` (alive, auth required), CT600 and CT101 both `running`; Mattermost's
|
||
`systemctl is-active` → `active`, `LISTEN *:8065`, `/api/v4/system/ping` → **HTTP 200**. Frontmatter
|
||
`verified_on` bumped to 2026-11-01. - **REGENERATED — `systems/skills-index.md`:** **301 unique skills** in 68 categories, **46 archived**. The count
|
||
is *unique `(category, skill_name)`* pairs, not `SKILL.md` file count — live `find` returns **302** files, and
|
||
the one difference is the known nested alias duplicate
|
||
`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` vs `productivity/obsidian-git-sync/SKILL.md`. **301 is correct — do not "fix" it to 302.** One skill was archived since the last run:
|
||
`hermes-runtime-worker-diagnostics` (46 archived, up from 45). - **FRESHNESS:** the 2026-11-01 report lists **1 stale / 113 fresh / 3 no-date** of 117 pages. The single stale
|
||
page (the memory-recall incident) has been corrected and re-verified above; the report was regenerated after
|
||
the fixes, per the script → fix → re-run → commit order. - **Probes (context, not findings):** `m93p` SSH over the tailnet timed out on the first two attempts and
|
||
succeeded on the third — a flaky path, not a host outage (`tailscale ping` → pong throughout). `homarr` on
|
||
CT201:8102 returns **307 → 200** on `/board`, so the dashboard page's "live, 200" claim still holds. CT201's
|
||
`:8443/` root returns **502** (upstream 8082 has no listener) and `:8443/mgr` returns **200** — both still as
|
||
the homepage-dashboard page records. No secrets, tokens, passwords or keys written. ## [2026-11-02] audit | Nightly review — the Weekly Review never ran, three workers killed by `-15`, and a red status with no row behind it
|
||
|
||
**Window:** 2026-11-01 00:00 → 2026-11-02 00:4x AWST. Live probes 00:05–00:55 on `state.db` (sessions),
|
||
`~/.hermes/cron/{jobs.json,executions.db,deliveries.db}`, `gateway.log` / `agent.log.1`, and both git trees. - **NEW — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`** (`status: active`):
|
||
the Sunday 18:00 `79f9bb41ecf5` closed `unknown` at 18:05:13 with *"Gateway connection failed; no local
|
||
fallback"*, **delivered nothing**, and its incident row `79f9bb_9c443b602eec` is **still `alerted`**. `deliveries.db` has **no row after 11:01:12**. Batch-mate `b63ed5817212` hit the identical error and
|
||
self-healed in 68 s. The same page records the **inverse** silent failure: `ecc1fc13655a` reads
|
||
`error`, `last_run 11-01T12:01:00`, `failure_streak: 1` — with **no execution row and no incident row**. - **NEW — `incidents/2026-11-01-cron-worker-sigterm-orphaned-claims.md`** (`status: resolved`): three
|
||
workers `SIGTERM`ed (`-15`) after adopting their claim (`85edf6772bca` 07:30:46; `96eb95ce12f2` +
|
||
`0e753e9720fd` 09:02:18) → *"whether side effects ran is unknown"*. **Fourth variant of the orphan
|
||
class, and the first with no gateway restart/reload/drain anywhere in the window** (PID `175269` up
|
||
since 10-31 19:24:40; the relay counter ran monotonic through both signals). Probable trigger: a
|
||
**30 s worker-ack window vs a ~70 s cold start** (`Plugin discovery complete: 83 found, 69 enabled`). All three incident rows `resolved`; one execution left `unknown`. - **UPDATED — `systems/scheduled-tasks.md`**: `## Count` advanced to the 2026-11-02 window (**167
|
||
`completed` / 1 `running` / 2 `unknown`** vs 161/4 on 10-31, 138/0 on 10-30, 177/0 on 10-29 — the
|
||
first non-clean day in four); fleet **29 jobs / 26 enabled / 3 disabled** unchanged; **unclosed
|
||
incident rows 1 → 1 but a different row** (all three rows the 10-31 audit left open closed on 11-01,
|
||
including **`0e753e_e7f8194a7d06`**, open since **2026-10-01** — 31 days); new `## Re-verified
|
||
2026-11-02` section covering the three failure modes, the 09:00 cold-start pair and the ledger. **Non-clean `last_status` 4 → 2** (`ea79a2107e36`, `79f9bb41ecf5`). - **UPDATED — `systems/messaging-integrations.md`**: Telegram degraded all day (dual-stack
|
||
`api.telegram.org` `ConnectTimeout` → IPv4-literal fallback; disconnect 20:51:53, polling restarts
|
||
22:33:30 / 23:12:23), and `getent hosts` currently returns **only an AAAA record** for
|
||
`api.telegram.org` — consistent with, but not proof of, a systematic timeout on the path DNS hands
|
||
the client. The **`relay` platform has had no adapter for ~29 h** (`Reconnecting relay` attempt
|
||
**147 @10-31 18:39:24 → 336 @11-02 00:10:47**, counter never reset — itself the proof of no gateway
|
||
restart). `066bb5191280`'s `thread_id 34795` home-DM miss is the **second confirmed variant** of the
|
||
thread-mismatch bug: the thread-less fallback fires, the content lands, and the **run is recorded
|
||
`failed` anyway**. Day's delivery health: **8 rows / 7 `delivered` / 1 `failed`**. - **UPDATED — `index.md`**: `updated`/`verified_on` → 2026-11-02, the audit summary line rewritten for
|
||
this window, and the two new incident links added to the incident list. - **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no decision reversed in the window. A full vault mirror pass was deliberately skipped (drift counts
|
||
below instead of a bulk copy) — an audit's mandate is smaller than a full sync. - **DRIFT (context, not findings):** the authoring tree was clean and in sync with Gitea
|
||
(`origin/master..HEAD` empty) before this audit; touched pages were mirrored into the vault's
|
||
existing conventions and hash-verified. **No secrets, passwords, tokens or keys written or quoted** —
|
||
every job id, PID, thread id and timestamp above is a non-sensitive operational identifier. ## [2026-11-03] audit | Nightly audit — 2026-11-02 window (first clean ledger day in five, and two ~9-hour failures it cannot see)
|
||
|
||
- **Scope:** `state.db` sessions spanning 2026-11-02 00:00 → 2026-11-03 00:2x AWST; `cron/executions.db`,
|
||
`cron/deliveries.db`, `cron_incidents`, `cron/jobs.json` (all read-only). Authoring tree
|
||
`/home/hermes/workspace/agent-estate-wiki` was **clean and in sync with Gitea** (`origin/master..HEAD`
|
||
empty) before this audit; `systems/skills-index.md` and `raw/stocktake/freshness-20261102.txt` were
|
||
**committed on the stocktake's behalf** (see below). - **NEW — `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md`:** the **third iteration-cap
|
||
in seven days** (10-27 tool-call budget, 10-28 cap, 11-02 cap) on `dca8482e4f76`
|
||
(**07:00:33 → 16:01:52, 9 h 01 m 19 s**), and **the first that raised no `cron_incidents` row at all** —
|
||
it closed `completed` / `error: NULL` / `delivery_outcome: suppressed`. Both Stage 2 slots exited
|
||
`[SILENT]`, so **nothing reached Notion, no cover letters were written and the 164 deduped candidates
|
||
were never persisted**. Page carries the **"green row, no artefact" detection table** contrasting the
|
||
three occurrences, and records the mid-run **URL-canonicalisation fix** (`au.seek.com` vs
|
||
`au.seek.com.au`) that had been silently dropping SEEK and Indeed entries from dedup. - **UPDATED — `incidents/2026-11-01-weekly-review-lost-to-gateway-handoff-failure.md`:** status
|
||
**`active` → `resolved`**. Live evidence: the **same execution row**
|
||
(`be6ec11a8e68428084af9ab18fb148db`) closed `completed` / `delivered` at **01:27:22** after
|
||
**7 h 26 m 43 s**, delivery at 01:27:21.802246, incident `79f9bb_9c443b602eec` auto-closed at
|
||
**01:27:21.606691**. The 18:05:13 `unknown` was a **reconciler timing artefact on a still-running job**,
|
||
not a verdict. Resolution section added; open questions 1–3 answered, **question 4 retained** (the two
|
||
unreconciled admission files `141448f9…json` / `69b5588f…json`). - **UPDATED — `systems/scheduled-tasks.md`:** `## Count` advanced to the 2026-11-02 window —
|
||
**177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows) vs 167/1/2 on 11-01, 160/4 on
|
||
10-31, 138/0 on 10-30; fleet unchanged at **29 jobs / 26 enabled / 3 disabled**; **unclosed incident
|
||
rows 1 → 0** (the first zero in the period); both 11-01 `unknown` rows resolved and `ecc1fc13655a`'s
|
||
ghost `error` cleared; deliveries **9 rows / 9 `delivered`**. New **`## Re-verified 2026-11-03`**
|
||
section documents the two invisible failures, the stocktake artefacts committed, the
|
||
`substr(started_at,1,10)` grouping rule and the **"do not read `running`/`unknown` as a verdict before
|
||
the expected runtime has elapsed"** correction. Frontmatter `updated`/`verified_on` → 2026-11-03,
|
||
`review_after` → 2026-11-10. - **UPDATED — `systems/browser-backend.md`:** `updated`/`verified_on` → 2026-11-03 plus a `## History`
|
||
entry for the **2026-11-02 REST-server wedge** (requests timing out with nothing reaching the service
|
||
log, cleared by `systemctl --user restart camofox-browser.service`; the four-board Job Radar run then
|
||
proceeded). **Live re-probe 2026-11-03 00:16:14 AWST:** unit `active`, `MainPID 953299`,
|
||
`ActiveEnterTimestamp Mon 2026-11-02 18:11:04`, `MemoryCurrent 37.5 MB`, `/health` → **200**
|
||
`{"ok":true,"engine":"camoufox","browserConnected":false,…}`, `/tabs` → `{"running":true,"tabs":[]}`. Page now states explicitly that this wedge is a **different fault from the 2026-10-12 memory ceiling**
|
||
(hang at 31 MB RSS vs exhaustion at 2.5 G), so the containment limits are not the applicable control. - **UPDATED — `index.md`:** `updated`/`verified_on` → 2026-11-03, the audit summary line rewritten for
|
||
this window, the new 11-02 incident link added, and the 11-01 Weekly Review entry annotated
|
||
**✅ RESOLVED 2026-11-02T01:27:21**. - **COMMITTED on the stocktake's behalf:** `1a8e90ea2271` finished with `systems/skills-index.md`
|
||
modified and `raw/stocktake/freshness-20261102.txt` untracked. Both are valid artefacts — freshness
|
||
**4 stale / 113 fresh** at a >21-day threshold, skills-index regenerated with **Devops 108 → 109**
|
||
(`pulse-turn-handling`) — so this audit committed them rather than let the next slot overwrite them. - **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||
decision reversed. A full vault mirror pass was deliberately skipped again (drift counts instead of a
|
||
bulk copy). - **DRIFT (context, not findings):** touched pages follow the vault's existing conventions and
|
||
SCHEMA.md frontmatter rules. **No secrets, passwords, tokens or keys written or quoted** — every job
|
||
id, PID, thread id and timestamp above is a non-sensitive operational identifier. ## [2026-11-04] audit — window 2026-11-03
|
||
|
||
**Verdict: one genuine red row, and the ledger produced it itself.** 2026-11-03 settled **170 `completed` /
|
||
1 `failed` / 1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since
|
||
10-31**. **New incident page:** `incidents/2026-11-03-job-radar-stream-active-timeout.md` — Job Radar Stage 1
|
||
(`dca8482e4f76`) ran **07:00:33.819190 → 13:28:52.305995 = 6 h 28 m 18 s** and died on
|
||
**`RuntimeError: stream active timeout`**. Its session (`local-7180cef7…`) compacted **five times** (08:04:35,
|
||
08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s** after its last tool result at
|
||
**12:23:28** before the harness recorded *"This turn did not complete."* at 13:28:49. Incident
|
||
`dca848_dfd4bea84f2a` (`failure_type: timeout`) opened 13:28:51.463929, **alerted** 13:28:55.141618, **still
|
||
open** → fleet unclosed rows **0 → 1**. The failure notice was delivered at **13:28:51.741724** and Pulse
|
||
carried it to Anthony (report `p34c8e899-1`, desktop session `20261031_190813_95fbde`: *"…dying on a stream
|
||
active timeout"*). **Updated:** `incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` — new 11-03 row in the
|
||
recurrence table plus a 2026-11-04 update section that **answers its open question 3**: output delivery
|
||
tracks the *terminal state*, not the run length. 11-02 ended `completed` → delivered nothing; 11-03 ended
|
||
`failed` → notice delivered and a row raised. Open question 2 narrows: the ledger **can** record a non-clean
|
||
state when the harness kills the turn; the iteration cap is the case that still escapes as `completed`. Question 4 is answered against us: **2026-11-02's candidates and 11-03's discovery are both unrecovered.**
|
||
|
||
**Second lost Stage-1 day:** both Stage 2 slots declined silently again (`5fc292ff0f3b` 11:30:36 → 11:32:16,
|
||
14:30:32 → 14:33:11) and `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30. **New page:** `systems/hermes-hq-push.md` — plugin **`hermes-hq-push` v0.9.27** (`kind: standalone`, author
|
||
*Hermes HQ*), tree mtime **2026-11-03 11:06**, **enabled** (`hermes plugins list` → `enabled`, state `user`; `config.yaml` `plugins.enabled` line 7556). Its own manifest states it **replaces `dispatch-push`**, and the
|
||
replacement is clean: **no `dispatch-push` reference survives in `config.yaml`** and no such directory
|
||
remains. It is a *different* stack from the Cadu/Vory bridge on `systems/push-notifications.md` (untouched,
|
||
still `degraded`). **Operational note recorded:** the plugin's `pre_tool_call` guard rejects any tool call
|
||
whose arguments contain the literal text of a launchd one-off-restart command — a wiki write quoting it was
|
||
blocked and had to be rephrased. **Fleet:** 29 jobs — 26 enabled / 3 disabled (unchanged). **Non-clean `last_status` 1 → 2:**
|
||
`dca8482e4f76` (`error`, streak 1) joins `066bb5191280` (unchanged `delivery_failed`). **Deliveries: 7 rows,
|
||
7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**, 13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271` 03:00:30.495724 → 10:41:13.851030 =
|
||
**7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing**, and again left its artefacts
|
||
uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt` untracked) — this
|
||
audit committed them. **Sessions in the window:** 16 (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop`
|
||
`20261031_190813_95fbde`) — 14 started, all `cron`. **No gateway restart:** PID **175269** is still the
|
||
2026-10-31 19:23:56 start (`hermes-dashboard` PID **176771**, 19:35:02). **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or repinned; no
|
||
decision reversed. **No secrets, passwords, tokens or keys written or quoted** — a Gitea remote URL with an
|
||
embedded credential was seen during verification and is not reproduced here ([REDACTED]). ## [2026-11-05] audit - window 2026-11-04
|
||
|
||
- **New page:** `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` - two 07:00 jobs
|
||
(`dca8482e4f76` **Daily Marketing Job Radar - Discovery**, `85edf6772bca` **Tech & AI Newsletter Digest
|
||
(Analytical)**) died **36 s apart** (09:41:44 / 09:42:20) on `RuntimeError: Upstream stream failed before
|
||
completion.` after **2 h 41 m** each, both on `custom:omni-101` / `smart-route`; both incidents still open
|
||
(`state = alerted`). Records the recovery: Stage 2 reconstructed the missing handoff from Stage 1's own
|
||
candidate files, re-verified every candidate live, and wrote **5 roles (board 479 -> 484) + 1 canonical
|
||
cover letter** (`MISSING_COVER_LETTER_COUNT 0`), consuming the handoff at 16:08:59. - **Updated `incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage.md`** - CT299 was frozen **twice**, not
|
||
once. Second freeze: a **manual** `vzdump 299 ... --storage ssd0` started **2026-11-04 08:30:04** and wedged
|
||
(`unexpected status`) until **11-05 ~00:30**. Also records the earlier clean failure at **07:36:44**
|
||
(log `vzdump-lxc-299-2026_11_04-07_36_44.log` on the dump target) and three `push_file` tasks from CT460
|
||
at 07:17-07:33, plus the abandoned `/mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299`. Live probe 2026-11-05
|
||
00:54:21 AWST: `LOCK: 0`, both `cgroup.freeze` flags `0`, guest `ALIVE`, `sftpgo: active`, 0 vzdump procs. - **Updated [[infrastructure/tailscale]]** - the sftpgo row now records recovery from *two* freezes. - **Updated [[systems/scheduled-tasks]]** - new ledger bullet: 11-04 settled **170 rows - 167 `completed` /
|
||
2 `failed` / 1 `running`**; non-clean `last_status` **2 -> 3**; unclosed incident rows **1 -> 3**; deliveries **8/8 `delivered`**; the two failures are one upstream signature; stocktake overran for the third
|
||
day (5 h 41 m 46 s, `free-stack` pin). - **Updated [[runbooks/job-radar]]** - cover-letter model route: `regenerate_cover_letter.py` defaults to
|
||
`CL_MODEL=deepseek-paid-smart` (line 82); the 11-04 run pinned `gemini/gemini-3.5-flash` after **HTTP 402**. `grep` shows **no `CL_MODEL` in `~/.hermes/.env`** - the override was process-local and did not persist, so
|
||
the next run inherits the default again. Endpoint probe 2026-11-05: `omniroute:20128/v1/models` -> **401**
|
||
and `100.78.63.20:20128/v1/models` -> **401** (live, key-gated) - so the route is reachable; the same model
|
||
serves other cron jobs successfully. Treat a 402 as route/time-specific until re-tested. - **Updated [[current-state]]** - new "Recent Changes (2026-11-05)" block; the page had been sitting at the
|
||
2026-10-30 block across five audits. - **Updated [[index]]** - new dated ledger paragraph, incidents list gains the new page, "Last updated" moved
|
||
2026-11-03 -> 2026-11-05. - **Live verification this audit:** pve (100.96.100.82) SSH probe at 2026-11-05 00:54:21 AWST (lock, cgroup
|
||
freeze flags, guest ALIVE, sftpgo active, proc count, task list, dump dir); SFTPGo reached end-to-end from
|
||
CT460 on `100.88.223.70:8080`; `jobs.json`, `executions.db`, `cron_incidents`, `deliveries.db` read live. - **Not done / unchanged:** no page removed or archived; no job added, removed, disabled or re-pinned; no
|
||
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
|
||
- *network-map.md re-verified (live probe 2026-11-05):* corrected the ct460 port table — **9377 is Camofox browser REST** (`/home/hermes/services/camofox-browser`), not Stratos; `8790` and `5174` are no longer listening; added `9120` (wiki serve), `9222` (Camofox CDP/WS), `7777` (OpenChamber), `6767` (Paseo), `8085` (AgentMail webhook), `9900` (gateway). Added the full Tailscale Serve table (12 routes on `:443`, `/` on `:9443` → 5173, Funnel `:8443` `/webhook` → 8085). Two routes are dead: `/agent-estate` → 8769 and `/synergy-prep` → 9123 have no listener. ## [2026-11-05] correction + live probe | fleet count, 11-04 ledger composition, `deepseek-paid-smart` dispatch
|
||
|
||
- **What was wrong.** The 2026-11-05 nightly-audit entry (written 00:5x AWST) recorded the fleet as **158 jobs
|
||
(144 enabled / 14 disabled)** and the 11-04 ledger as closing **1 `item_suppressed`**. Both were wrong. Live
|
||
re-reads - `~/.hermes/cron/jobs.json`, `hermes cron list`, and the in-flight job count - each give **29 jobs /
|
||
26 enabled / 3 disabled**; `executions.db` shows 11-04 at **170 rows - 167 `completed` / 2 `failed` / 1
|
||
`running`** (this audit), with **no `item_suppressed` status anywhere on 11-04**. - **Where corrected:** `systems/scheduled-tasks.md` (Count bullet + correction clause),
|
||
`current-state.md` (11-05 block), `index.md` (11-05 ledger paragraph), and this entry. No other page carried
|
||
the bad number. - **11-03 re-settle:** `171 completed / 1 failed` (172 rows) - its `running` row has since completed - so 11-03
|
||
and 11-04 are the only consecutive days with a `failed` row since 10-31. - **Live probe - model dispatch, 2026-11-05 01:33:12 AWST (first-hand, keyed):**
|
||
`POST http://omniroute:20128/v1/chat/completions` with `deepseek-paid-smart` -> **HTTP 503**
|
||
`ALL_TARGETS_SKIPPED` (`poolSize: 5`, `attempted: 0`). The same request to
|
||
`http://100.78.63.20:20128/v1/chat/completions` (`omniroute-1`) -> **HTTP 200**, `model:
|
||
deepseek/deepseek-v4-flash`. Unkeyed `/v1/models` on both hosts returns **401** - auth-gating, not health. The
|
||
fleet's only `deepseek-paid-smart` pin is **this audit job** (`cbfe434a3b5e`, provider `custom:omni-101`, i.e. the `omniroute-1` node), which is why the audit still completes while a direct CT600 probe 503s. - **Re-verified and matching (no change needed):** `cron_incidents` - 3 unclosed rows, all `state = alerted`; `deliveries.db` - 8 rows on 11-04, **8/8 `delivered`**, including both `for_failure` notices (09:41:43,
|
||
09:42:20); non-clean `last_status` **3**; pin distribution **14 `smart-route` / 6 `free-stack` / 8 unpinned /
|
||
1 `deepseek-paid-smart`**. - **Files touched this pass:** `systems/scheduled-tasks.md`, `current-state.md`, `runbooks/job-radar.md`
|
||
(`CL_MODEL` caveat now carries the keyed probe), `index.md`, `log.md`. Frontmatter `updated` / `verified_on`
|
||
bumped to 2026-11-05 where stale. - **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
|
||
decision reversed. **No secrets, passwords, tokens or keys written or quoted** - the API key was read from
|
||
`~/.hermes/.env` into the shell for the probe only. ## [2026-11-05] stocktake | Skills-index generator divergence FIXED at the root; Docker + Tailscale drift corrected against live probes
|
||
|
||
- **Ran `wiki-maintain.sh`** (repo canonical path) - exit 0. Regenerated `systems/skills-index.md`
|
||
(**302 skills / 68 categories**, `.archive/` excluded, 46 archived), wrote
|
||
`raw/stocktake/freshness-20261105.txt` (**0 stale / 122 fresh / 3 no-date of 125 pages**), rebuilt
|
||
Quartz, deployed to `public/`, restarted `hermes-wiki-serve.service`. - **ROOT CAUSE FOUND AND FIXED - the `systems/skills-index.md` flip-flop had TWO WRITERS.** The repo
|
||
`wiki-maintain.sh` (mtime 2026-11-03) and `~/.hermes/scripts/wiki-maintain.sh` (mtime **2026-08-17**) are
|
||
**different scripts**, each with its own generator: the repo one scans with `find` excluding
|
||
`*/.archive/*` (-> **302 skills / 68 categories**); the stale one uses `Path.rglob('SKILL.md')` and a
|
||
2-segment category heuristic (-> **349 skills / 26 categories**, because it counts the parked archive). `wiki-maintenance/SKILL.md` line 569 pointed the **23:59 nightly audit** at the stale path while the
|
||
**03:00 stocktake** ran the repo path - so each job overwrote the other's page and the wiki oscillated
|
||
nightly with **no error raised by either job**. - **Fix:** `~/.hermes/scripts/wiki-maintain.sh` is now a **delegating wrapper** with **no generator logic**
|
||
- it `exec`s the repo script. The repo script gained **`--no-build`** so the wrapper honours the
|
||
interface its callers were written against without carrying a second copy of the generator. - **Verified by execution, not inspection:** `bash ~/.hermes/scripts/wiki-maintain.sh --no-build` ->
|
||
`Written ... with 302 skills in 68 categories`; both scripts pass `bash -n`. - **SKILL.md** repointed at the repo path, with a note that a generator must live in exactly one place. -> [[incidents/2026-11-05-skills-index-generator-divergence]]; the "still open" item on
|
||
[[incidents/2026-08-15-skills-index-drift]] is closed. - **`infrastructure/docker-services.md` corrected against live `docker ps` (all hosts, 2026-11-05 05:2x):**
|
||
- CT200 **61 running / 68 total** (was "62"); the **`worldmonitor-seeder-scheduled`** entry recorded on
|
||
2026-11-04 as "unhealthy" **does not exist on any host** - `docker ps -a --filter name=seeder` across
|
||
CT200/201/333/460 returns **zero** matches. Live CT200 shows exactly three worldmonitor containers
|
||
(`worldmonitor` `Up 5 days` **healthy**, `worldmonitor-redis`, `worldmonitor-redis-rest`). There is **no
|
||
seeder container and no worldmonitor-seeder cron job** anywhere in the estate. - CT333 first container renamed again: `nostalgic_leavitt` -> **`flamboyant_mahavira`** (image
|
||
`ghcr.io/github/github-mcp-server`, created 2026-11-04T10:53:53Z). CT333 = **8 running / 9 total**. - CT201 **20 running / 26 total**; CT460 **1 running / 1 total** (`ignis` `Up 5 days`); CT1001
|
||
**1 running / 9 total** (`sshwifty`). Per-host "Last verified 2026-11-05" added. - **`infrastructure/tailscale.md` re-verified against live `tailscale status --json`:** the node table is
|
||
**exact** - 52 rows, 52 live IPv4 peers, **no live peer missing from the table, no table row without a live
|
||
peer, and every Online/offline state matches** (checked programmatically, not by eye). Two offline ages
|
||
corrected: `conduit` 59d -> **60d**, `ubuntu-server-xfce` 75d -> **76d**. `tailscale serve status` still
|
||
matches the recorded table exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443`
|
||
tailnet route). - **`infrastructure/hosts.md`:** skills figure corrected to **302 skills / 68 categories** (was "69
|
||
categories"); `updated` / `verified_on` -> 2026-11-05. - **`index.md`:** skills-index bullet refreshed to the 2026-11-05 figures (302 / **68 categories**,
|
||
freshness **0 stale / 122 fresh / 3 no-date of 125**), the `_archive/` typo corrected to `.archive/`, and
|
||
the new incident linked. - **Freshness:** the 8 pages stale on 2026-11-04 were all re-verified and refreshed, so the 2026-11-05 report
|
||
reads **0 stale**. The 3 pages with no `updated` date are `SCHEMA.md`, `log.md` and
|
||
`experiments/active/example.md` - `log.md` is append-only by design, the other two are templates. - **Files touched this pass:** `systems/skills-index.md`, `infrastructure/docker-services.md`,
|
||
`infrastructure/tailscale.md`, `infrastructure/hosts.md`, `index.md`,
|
||
`incidents/2026-11-05-skills-index-generator-divergence.md` (new),
|
||
`incidents/2026-08-15-skills-index-drift.md`, `wiki-maintain.sh`, `log.md`. - **Not done / unchanged:** no job added, removed, disabled or re-pinned; no page removed or archived; no
|
||
decision reversed. **No secrets, passwords, tokens or keys written or quoted.**
|
||
|
||
## [2026-11-06] audit - window 2026-11-05
|
||
|
||
- **Window reviewed:** 2026-11-05 00:00-24:00 AWST; live reads 2026-11-06 00:1x-01:0x AWST. Fleet unchanged at
|
||
**29 jobs (26 enabled / 3 disabled)** - a fourth consecutive audit at the same count, no job added, retired,
|
||
disabled or re-pinned. - **New page: [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (status: OPEN).** Three live
|
||
`.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit in
|
||
plaintext in **17 rows of `~/.hermes/mnemosyne/data/mnemosyne.db`** - `memories.content` 4, `episodic_memory.content`
|
||
3, `working_memory.content` 1, `gists.text` 4, `memoria_facts.context_snippet` 1, FTS mirrors 4 - and
|
||
`CAMOFOX_API_KEY` sits verbatim in the `prompt` of two **enabled** Facebook Birthday jobs (`0e753e9720fd`,
|
||
`fad5a86979d4`) and hence in `cron/deliveries.db`. Method: exact-substring match of each `.env` value across
|
||
**every TEXT/BLOB column of every table** in `mnemosyne.db` plus its FTS mirrors; byte-membership scans of the
|
||
10 `.db` files under `~/.hermes/config-backup/` (six are 0-byte; only `cron/deliveries.db` carries any tested
|
||
value), the shared-surface DB, the seven uncompressed rotation snapshots under `~/.hermes/mnemosyne/backups/`
|
||
(all seven reproduce the DB in full) and a chunked scan of the **1.7 GB session store `state.db`**, which
|
||
carries all four values and is the likely capture origin (capture writes verbatim session text). **Nothing was
|
||
rotated, deleted or scrubbed; no credential value was printed, logged or written to any page.**
|
||
- **Both 11-04 casualties recovered in full.** `85edf6772bca` (*Tech & AI Newsletter Digest*) completed its 07:00
|
||
run in **7 m 01 s** (07:00:42 -> 07:07:43) and `dca8482e4f76` (*Daily Marketing Job Radar - Discovery*) in
|
||
**5 h 35 m 35 s** (07:00:42 -> 12:36:18); neither re-raised `RuntimeError: Upstream stream failed before
|
||
completion.` The 11-05 ledger settled **175 rows / 18 jobs - 173 `completed` / 2 `running` / 0 `failed`**, the
|
||
**first zero-`failed` day since 2026-11-02**; the two `running` rows are `ea79a2107e36` (*Mnemosyne Health
|
||
Monitor*, 12:00:45 slot, still in flight 12 h+ with no delivery) and this audit (`cbfe434a3b5e`). Deliveries
|
||
**10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`,
|
||
*Uber Eats Monthly Spend Report*, `delivery_failed`, streak 0). - **Incident ledger emptied.** Unclosed rows **3 -> 0**: the digest row closed **07:07:42** and both Job Radar rows
|
||
(`dca848_dfd4bea84f2a` `timeout`, `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. `incidents/2026-11-03-job-radar-stream-active-timeout.md` -> `status: resolved` with a Resolution section; `incidents/2026-11-04-upstream-stream-failure-0700-cohort.md` closed out - the first empty ledger since
|
||
2026-11-02. - **Updated [[systems/scheduled-tasks]]** - frontmatter (`updated` / `verified_on` -> 2026-11-06, `review_after`
|
||
2026-11-07, `live-sessions-db` added to `sources`), a new `## Count` bullet for the 2026-11-06 read and a new
|
||
`## Re-verified 2026-11-06` span (recoveries, zero-`failed` ledger, delivery ledger, the secrets finding). - **Updated [[systems/mnemosyne]]** - frontmatter dates -> 2026-11-06 (`review_after` 2026-11-13), `sources` +
|
||
`live-secrets-scan, live-state-db`, and a **security entry at the head of `## Known Issues`** recording that the
|
||
store's own tables carry 17 rows of live credential material, that recall is the exposure path, and that only
|
||
rotation covers the copies already taken. - **Updated [[current-state]], [[index]]** (`## Incidents` link for the new page, header summary rewritten to the
|
||
2026-11-06 read) and this log. - **Freshness:** `raw/stocktake/freshness-20261106.txt` (re-run after this audit's writes) -> **0 stale (>=90d),
|
||
124 fresh, 3 no-date of 127 pages**, the new incident page included. The three no-date pages are unchanged (`SCHEMA.md`,
|
||
`experiments/active/example.md`, `log.md`). - **Not done / unchanged:** no page removed or archived; no decision reversed; `066bb5191280` and the
|
||
`deepseek-paid-smart`-on-`omniroute` dispatch fault carried, not re-diagnosed. **No secrets, passwords, tokens
|
||
or keys written or quoted** - the incident page records where secrets live, never their values. ## 2026-11-06 03:0x AWST - stocktake (wiki-stocktake-maintenance)
|
||
|
||
- **Ran `wiki-maintain.sh` (repo script, canonical path).** `systems/skills-index.md` regenerated ->
|
||
**302 skills / 68 categories** (unchanged count; date rolled to 2026-11-06). Freshness audit at the
|
||
**21-day** threshold -> **0 stale / 124 fresh / 3 no-date of 127 pages**. Quartz rebuilt (761 files emitted,
|
||
127 processed) and deployed to `public/`; `hermes-wiki-serve.service` restarted. Script exit 0. ⚠ **Threshold correction:** the previously committed `freshness-20261106.txt` carried a **90-day** threshold
|
||
(the nightly audit's setting); the stocktake script's threshold is **21 days**, and this run's report is the
|
||
authoritative one for today. - **Freshness: 0 stale pages.** The three no-date pages are unchanged (`SCHEMA.md`,
|
||
`experiments/active/example.md`, `log.md`). No page is over 21 days. - **Live-vs-page drift diff (the age scan alone would have missed all of this).**
|
||
- **[[infrastructure/tailscale]] - 1 state transition + 8 offline-age drifts + 2 dead Serve routes.**
|
||
`anthonys-mac-mini` **came back online** (peer record `Online: true`; `tailscale ping 100.76.60.37` ->
|
||
`pong ... via DERP(syd) in 103ms`). Offline ages refreshed +1d each: cloudreve 103->**104d**,
|
||
elementsynapse 106->**107d**, freshrss-new 79->**80d**, octoagent 3->**4d**, octop 1->**2d**,
|
||
omarchybook 14->**15d**, omarchymac 3->**4d**, openmausbot 14->**15d**. **Every peer IP still matches the
|
||
live JSON exactly; no live peer missing; no orphan row.** `m93p` re-pinged -> pong (3ms); `octop`
|
||
re-pinged -> timeout, matching its `LastSeen`. Container states cross-checked against live `pct list` on
|
||
**both** hypervisors (pve 17 guests, m93p 19 guests + 2 VMs). ⚠ **`/agent-estate` -> 8769 and
|
||
`/synergy-prep` -> 9123 are DEAD** (no listener) while still present in `tailscale serve status` - both
|
||
routes now marked DEAD on the page, matching [[infrastructure/network-map]] which had already recorded
|
||
them as dead on 2026-11-05. Every other Serve target has a live listener. - **[[infrastructure/docker-services]] - the 2026-11-05 "the seeder does not exist on any host" finding was
|
||
FALSE; corrected on the page with strikethrough + dated correction.** The `worldmonitor-seeder-*`
|
||
containers are **transient and systemd-timer driven** on CT200 (`worldmonitor-seed.timer`,
|
||
`OnUnitInactiveSec=4h` -> `worldmonitor-seeder-scheduled`; `worldmonitor-seed-fast.timer`,
|
||
`OnUnitInactiveSec=15min` -> `worldmonitor-seeder-fast`), so they are visible only while a run is in
|
||
flight. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x **neither** was - hence
|
||
the false negative. The "no worldmonitor-seeder cron job" claim was also wrong: the driver is a **systemd
|
||
timer pair, not cron**. The earlier "rename" narrative was wrong too: **one script, two timer-driven
|
||
container names**. CT200's 61-container running set still matches live **set-wise** (no drift). **CT333's
|
||
container renamed again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z). CT201 20/26, CT460 1/1, CT1001 1/9, CT333 8/9 - all unchanged and correct. - **[[systems/scheduled-tasks]] - no drift.** Live `jobs.json` reads **29 jobs / 26 enabled / 3 disabled**,
|
||
**1** non-clean `last_status` (`066bb5191280` Uber Eats Monthly Spend Report, `delivery_failed`, streak 0) -
|
||
exactly what the page already records. - **[[infrastructure/hosts]] - cron count 29/26/3 still correct** (last re-verified 2026-10-31). - **Updated [[current-state]] and [[index]]** - the false seeder claim and the "every state matching" Tailscale
|
||
claim are now marked superseded with dated corrections rather than silently rewritten (SCHEMA rule 9). - **Not done / unchanged:** no page removed or archived; no decision reversed; the open
|
||
`deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not
|
||
re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or
|
||
quoted.**
|
||
|
||
- **Follow-up (04:0x AWST): the one page sitting exactly on the 21-day threshold was re-verified rather
|
||
than left to expire.** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` was at **21d** and
|
||
would have tripped the threshold tomorrow. Its residual claim ("`omniroute-1` answers again") was
|
||
re-probed **live** and still holds: TCP connect to `100.78.63.20:20128` succeeds (no `Errno 111`),
|
||
the Tailscale peer reads `Online: true`, and a keyed `POST /v1/chat/completions` returns **HTTP 200**. Page re-verified (`updated`/`verified_on` -> 2026-11-06, `review_after` -> 2026-12-06) with the probe
|
||
recorded. ⚠ **New trap recorded on the page and in [[systems/scheduled-tasks]]:** on `omniroute-1` the
|
||
**raw** model id `deepseek/deepseek-v4-flash` returns **HTTP 401 `No active credentials for provider:
|
||
deepseek`**, while the fleet's **`deepseek-paid-smart` alias returns HTTP 200** and resolves to that same
|
||
model. The 401 is an **alias-routing artefact, not an outage** — probe the alias the fleet pins, never the
|
||
raw model id. (A first probe with the wrong `key_env` also produced a 401; the correct key for
|
||
`omni-101` is `HERMES_CUSTOM_100_78_63_20_20128_API_KEY`.)
|
||
|
||
## [2026-11-07] audit - window 2026-11-06
|
||
|
||
**Headline: a red row with no alert.** 2026-11-06 settled **172 execution rows across 16 distinct jobs - 170
|
||
`completed` / 1 `failed` / 1 `running`** (this audit). The single failure is a **new, non-restart** signature:
|
||
the Job Radar's **11:30 slot** (`5fc292ff0f3b`, row `95347fa5…`, 11:30:32 -> 13:26:44) was **discarded as
|
||
stale** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job
|
||
(08:27:09 -> 13:26:22, 4 h 59 m 13 s)** was still holding the claim and delivered first. Every previous
|
||
occurrence of this string came from a restart/reload orphaning a live run; **this one had no restart** - the
|
||
claim was lost to another fire of the same job. The failed row raised **no `cron_incidents` row** (newest
|
||
ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted**. New page ->
|
||
[[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. - **Ledger/day:** deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; fleet **29 jobs (26 enabled / 3
|
||
disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), the
|
||
**fifth consecutive audit at the same count**, nothing added/retired/disabled/repinned, fleet `updated_at`
|
||
**2026-11-07T00:39:46+08:00**. Non-clean `last_status` unchanged at **1** (`066bb5191280`,
|
||
`delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0**; the one open page remains
|
||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] (**untouched in-window** - no
|
||
rotation, deletion or scrub). - **Job Radar recovered the day in full:** the 14:30 slot ran a normal **12 m 27 s** cycle and delivered at
|
||
14:42:59 - `SAME_DAY_STRONG_COUNT 4`, `VERIFIED_COVER_LETTER_COUNT 4`, `MISSING_COVER_LETTER_COUNT 0`,
|
||
cache total **494**, 10 today-dated rows, 20 non-empty artifacts, *Run status: COMPLETE*. - **The stocktake's overrun streak fully unwound:** `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m
|
||
03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Stage 1 (`dca8482e4f76`) ran
|
||
clean for a **second consecutive day** (07:00:44 -> 08:29:21, 1 h 28 m 37 s, `suppressed`), so the 11-04
|
||
Open entry about its two lost candidate sets is **cleared in substance** - the 11-03 ledger row closed
|
||
`resolved` at 2026-11-05T12:36:17 and both runs since have been clean. - **The last audit's open question answered itself:** the Mnemosyne Health Monitor's 11-05 12:00:45 slot
|
||
(still in flight 12 h+ with no delivery at the last audit) **completed at 2026-11-06T00:14:39 after 12 h
|
||
13 m 54 s and delivered**; its 11-06 12:00 slot then took **7 h 47 m 56 s** and delivered too. Ballooned on
|
||
consecutive days, never lost, no incident row - **watching, not open.**
|
||
- **Sessions:** **15 cron sessions started in-window (495 messages / 273 tool calls)**, 16 sessions with
|
||
activity (incl. one `desktop` session, 146 messages). Busiest: `wiki-stocktake-maintenance` 157, Job Radar
|
||
Discovery 92, `Enhanced Weekend Adventure Planner` 82, Job Radar Write and Letters (14:30) 28, Mnemosyne
|
||
Health Monitor 27. - **Wiki maintenance:** `wiki-maintain.sh` run **after** this audit's writes (freshness report
|
||
`raw/stocktake/freshness-20261107.txt`); skills-index regenerated; Quartz rebuilt. Pages touched:
|
||
[[systems/scheduled-tasks]] (+frontmatter, +Count bullet, +`Re-verified 2026-11-07` section),
|
||
[[current-state]] (+frontmatter, +Recent Changes block), this index (+frontmatter, +summary, +incident
|
||
bullet), [[runbooks/job-radar]] (+concurrency subsection), this log, and the new incident page. Vault
|
||
mirror `/home/hermes/wiki` updated (both conventions). ## 2026-11-07 03:0x AWST - stocktake (wiki-stocktake-maintenance)
|
||
|
||
- **Script:** `wiki-maintain.sh` (repo, canonical) run clean — skills-index regenerated (**302 skills /
|
||
68 categories**), freshness report written to `raw/stocktake/freshness-20261107.txt`, Quartz rebuilt
|
||
(**769 files**) and deployed to `public/`, `hermes-wiki-serve.service` restarted. - **Freshness:** **0 stale, 125 fresh, 3 no-date of 128 pages** — no page is past the 21-day threshold. (`log.md` has no `updated` date and lands in the no-date bucket, which is correct for an append-only log.)
|
||
**Freshness is not accuracy**, so a live-vs-page diff was run against every page carrying live claims. - **Tailscale (infrastructure/tailscale.md):** table rebuilt from `tailscale status --json`. **1 state
|
||
transition — `octop` (`100.92.163.122`) is back online** (peer record `Online: true`; `tailscale ping` →
|
||
`pong from octop via DERP(syd) in 98ms`); it is still **no LXC or VM on either hypervisor**, so it is an
|
||
orphaned tailnet peer, not an estate change. **9 offline ages refreshed (+1d each).** All **51 live IPv4
|
||
peers** match the table's IPs exactly — **no missing peers, no orphan rows, no IP mismatches** (52 rows =
|
||
51 peers + `hermes`; the ~23 IPv6-only `funnel-ingress-node` peers excluded by design). The CT460 Serve
|
||
route table still matches `tailscale serve status` **exactly (12 paths on :443)**; **both DEAD targets
|
||
re-confirmed dead** (no listener on `8769` `/agent-estate`, none on `9123` `/synergy-prep`), every other
|
||
Serve target has a live listener, and `/gitea` (`100.82.100.128:3000`) answers **HTTP 200**. - **Proxmox (infrastructure/proxmox.md):** ⚠ **`m93p` runs 20 LXCs, not 19 — CT107 `mem0` was missing**
|
||
(running; `mem0-mcp.service` active on `:8890`; `pct exec 107 -- tailscale ip -4` → `100.73.94.53`). Added to the table; the 2026-10-31 reading of 19 was already short for the same reason. Live now reads
|
||
**20 LXCs — 18 running, 2 stopped** (CT998, CT1264). `pve` still matches exactly: **17 LXCs + 4 VMs**. Both hosts run the identical **`pve-manager 9.2.10`** build. - **⚠ `m93p` uptime claim was wrong.** The page read *"up 1 day, 12:02 … rebooted ~2026-10-30 16:0x"*; live
|
||
`uptime -s` shows the host **booted 2026-11-05 21:19:52** (up **1 day, 7:18**, load **0.76 / 1.08 / 1.24** — normal). The 2026-10-30 reboot was real but belonged to **`pve`** (boot **2026-10-30 16:24:00**). - **Storage refreshed on both hosts** against live `pvesm status`: `pve` `ssd0` **76.43 → 78.29 %**
|
||
(~152 GiB free), `SSD1` **64.32 → 65.46 %**, `fred-qnap` **82.77 → 84.21 %**, `local` **79.66 → 81.62 %**,
|
||
`local-lvm` **13.75 → 14.25 %**; `m93p` iMac shares + `local` **60.25 → 76.21 %**, `nfs-proxmox-*`
|
||
**82.78 → 84.21 %**, `nfs-ssd0-39` **76.43 → 78.29 %**, `omb-internal` **57.92 → 64.66 %**. - **Docker (infrastructure/docker-services.md):** set-wise diff against live `docker ps` on CT200/201/333 —
|
||
CT200 **61 running / 68 total** exact, CT201 **20 / 26** exact, CT333 **8 / 9** exact, **but CT333's
|
||
enumerated list still opened with `flamboyant_mahavira`, which is not running** (live name
|
||
**`nostalgic_keldysh`**, created 2026-11-05 22:15:57). Corrected. CT460 `ignis` running. - **Cross-page corrections:** [[infrastructure/network-map]] `/gitea` was annotated *"Gitea (ct207 tailnet
|
||
IP)"* — it is **CT221 on `m93p`** (ct207 is `100.82.40.23`); also its `m93p` PVE version read *9.2.0*
|
||
(both hosts are 9.2.10). [[agents/hermes-chaos]] read *"`v0.22.3` … `5b147ab036d6` — the same build as
|
||
CT460"*, which contradicted itself: live is **`v0.22.5` … `bf0203a794a0`**, now genuinely identical to
|
||
CT460's (`hermes update --check` reports **v0.23.0** available). [[agents/hermes-production]]'s
|
||
*"2 cron jobs still pin `custom:omni`"* is **no longer true** — live `jobs.json` has **0** on the bare key
|
||
and **21** on `custom:omni-101`. - **Fleet re-verified unchanged:** CT460 **29 jobs (26 enabled / 3 disabled)**; CT450 **25 jobs
|
||
(13 enabled / 12 disabled)** with **9 shared names** — confirmed by a live set-diff of both `jobs.json`. A keyed `deepseek-paid-smart` chat probe against `omniroute-1` (`100.78.63.20:20128`) returns **HTTP 200**; `config.yaml` still defaults to `custom:omni-101`. - **Pages touched:** [[infrastructure/tailscale]], [[infrastructure/proxmox]], [[infrastructure/hosts]],
|
||
[[infrastructure/network-map]], [[infrastructure/docker-services]], [[agents/hermes-production]],
|
||
[[agents/hermes-chaos]], [[current-state]], [[index]], this log. No new incident page — no incident was
|
||
found; every drift above was a documentation drift corrected against a live probe. ## 2026-11-07 05:1x AWST - incident resolution: Gitea config backup push dead since 2026-08-08
|
||
|
||
- **Anthony's ask:** "fix this" on the failing backup alert — with a pointer to check Notion for the API key. - **[[incidents/2026-08-08-gitea-config-backup-push-dead]] is now `status: resolved`.** The nightly
|
||
`e27da6c34bbb` config backup had been committing locally and failing to push **every night since
|
||
2026-08-08** — **92 commits** stranded — while its cron row read `last_status: ok`. - **Four independent root causes, all confirmed against live systems:**
|
||
1. **No Gitea credential.** `origin` was HTTPS with no userinfo; the global `credential.helper=store`
|
||
file `~/.git-credentials` held only a `github.com` entry. The token in the Notion vault was
|
||
**valid** and byte-identical to the one the wiki repo carries in its remote URL — it simply was
|
||
never wired into the backup repo. Fixed with a host-scoped `git credential approve` entry (not
|
||
embedded in the URL); `git ls-remote origin` now authenticates with `GIT_TERMINAL_PROMPT=0`. 2. **`backup.sh` had no `exit` statement at all** — it always returned 0, so the scheduler's
|
||
`success = (returncode == 0)` recorded `ok` on every failed night. The scheduler was correct; **the script was the liar**. This is why the outage ran 92 days without a job-level red. 3. **A 1.76 GB `checkpoints/` backlog** in the unpushed range (13 blobs > 100 MB) — junk that
|
||
`.gitignore` already excluded for *new* commits but that history still carried. 4. **Both secondary legs broken.** The `github` remote did not exist, and the fallback pushed a
|
||
branch's **full ancestry**, which contains a 7.45 GB tarball blob — so GitHub's 100 MB file
|
||
limit could never accept it. The AgentMail alert called a `python3 send_email.py` that no longer
|
||
exists, via a `chaos-agent@agentmail.to` inbox that no longer exists. - **Backlog drained:** `git filter-repo --partial --refs origin/main..main --path checkpoints
|
||
--invert-paths --prune-empty never --force` stripped `checkpoints/` from the **unpushed range only**
|
||
(`origin/main` untouched, push stayed a fast-forward), then `checkpoints/` was untracked at the tip. Push set **~2 GB → 264.8 MB**; 93 commits pushed in 11.6 s; the 7.45 GB tarball was already on
|
||
`origin/main` and never resent. Pre-rewrite history preserved under tag
|
||
`backup/pre-rewrite-20261107-032658`. **Token leakage scan over the HEAD tree, the working tree and
|
||
every unpushed blob: clean.**
|
||
- **Script rewritten** (`~/.hermes/config-backup/backup.sh`, via OpenCode + a correction pass): exits
|
||
**0 only on verified success**; stamps `.last_backup_ok` **only** after remote verification; decides
|
||
the push by **remote-vs-local** rather than by working-tree changes (the exact mechanism that created
|
||
the backlog); guards every `git ls-remote` with `|| true` so an unreachable remote reaches
|
||
`notify_failure` instead of dying at exit 128 *before* the alert; pushes GitHub as a **parentless
|
||
orphan snapshot** (`git commit-tree` from `HEAD^{tree}` — no history, no >100 MB blob, HEAD/branch/
|
||
index untouched) as a warning-only secondary target; and reports Telegram/AgentMail delivery
|
||
**honestly** (`curl --fail` + `"ok":true`; the real `agentmail` CLI, key read from `.env` at runtime
|
||
and passed via the environment, never argv). - **Verified live 2026-11-07:** real run exit **0** in 47 s; `git rev-parse HEAD` ==
|
||
`git ls-remote origin refs/heads/main` == `a50a5170c1c0e2af5235fa63a95a732e2d780703`; `rev-list --count origin/main..main` = **0**; Gitea API `main` = `a50a5170c1c0`, updated
|
||
`2026-11-07T05:10:46+08:00`; `.last_backup_ok` refreshed (was stale from 2026-11-06 11:00). GitHub
|
||
mirror now carries a **0-parent** orphan snapshot `faed9fb5c677` (30,086 files, largest blob 23.5 MB)
|
||
with `master` untouched at `c99c94e6b13b` — **no data lost**, since GitHub's tree was a strict subset
|
||
of local HEAD (only `BACKUP-INDEX.md` added). AgentMail and Telegram alert paths each exercised
|
||
directly (AgentMail self-test to the agent's own inbox; Telegram `getMe`, no message sent to Anthony). Sandbox E2E: unreachable remote → exit 1 + alert + health **not** stamped; local-ahead-with-clean-tree
|
||
→ pushed + verified; already-up-to-date → no push, exit 0. - **Pages touched:** [[incidents/2026-08-08-gitea-config-backup-push-dead]] (status → resolved, full
|
||
Resolution section added), [[infrastructure/backups]] (⚠ → ✅, re-verified 2026-11-07),
|
||
[[systems/scheduled-tasks]] (job row → RESOLVED), this log. - **Note — unrelated pre-existing state:** the 03:00 `wiki-stocktake-maintenance` slot **failed** with
|
||
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
|
||
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
|
||
included in this commit so the tree is not left dirty. ## [2026-11-08] audit - window 2026-11-07
|
||
|
||
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
|
||
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
|
||
0 `for_failure`. - **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
|
||
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
|
||
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
|
||
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
|
||
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
|
||
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
|
||
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
|
||
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
|
||
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
|
||
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13). - **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
|
||
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
|
||
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52. - **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`); the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
|
||
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
|
||
`pid 3109529`). CT450's install stamp now also reads **0.23.0**. - **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
|
||
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
|
||
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
|
||
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
|
||
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
|
||
an auth error. - **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
|
||
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
|
||
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed. - ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
|
||
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
|
||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
|
||
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
|
||
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
|
||
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy). - **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`. - **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
|
||
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
|
||
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
|
||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
|
||
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
|
||
(`/home/hermes/wiki`). - **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
|
||
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
|
||
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
|
||
**2026-11-08 00:38:29 AWST**. - **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script) run **after** this audit's writes —
|
||
`systems/skills-index.md` regenerated (**299 skills / 67 categories**; disk shows 300 `SKILL.md`, one nested
|
||
duplicate de-duped by the generator), freshness report `raw/stocktake/freshness-20261109.txt`
|
||
(**0 stale / 127 fresh / 3 no-date of 130 pages**), Quartz rebuilt (**130 markdown → 793 files** deployed to
|
||
`public/`) and `hermes-wiki-serve.service` (a **user** unit — `systemctl --user`) restarted
|
||
**2026-11-09 00:18:30 AWST**. Published pages verified **HTTP 200** at `http://127.0.0.1:9120/<path>` **and**
|
||
on the external route `https://hermes.kangaroo-eel.ts.net/wiki/<path>`. - **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
|
||
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
|
||
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
|
||
the upstream error code only. ## [2026-11-09] nightly audit | Window 2026-11-08 — a 30 s hook budget, not a schedule, ate the day
|
||
- **New page:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` — **RESOLVED same day.**
|
||
- **The finding:** `conduit_push` is the **only** `pre_tool_call` registrant on the host
|
||
(`plugins/conduit_push/__init__.py:119`), and its callback overran Hermes's built-in **30 s** hook budget
|
||
**58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** — `~/.hermes/logs/errors.log`
|
||
holds **12** lines on 11-07 and **46** on 11-08, plus **89** *"skipped after previous timeout or while
|
||
still running"* suppression lines. Every tool call degraded from slow to suppressed. - **The fix (2026-11-08 13:35 AWST):** `plugins.hook_callback_timeout` **30 → 120** in
|
||
`~/.hermes/config.yaml`. **Verified live**, and **verified effective** — the last timeout line is still
|
||
08:21:03, re-checked at audit time. - **A plausible diagnosis measured and rejected:** the plugin's caching was *not* the fault. Timed
|
||
in-process: **3.1 s cold, 0.00 s warm** (`session_titles.py:21 CACHE_SECONDS = 30.0`). The
|
||
*"re-opens a 1.87 GB `state.db` per call, ~9 s each"* theory was **wrong**; the trigger is **load** on the
|
||
2-core CT460 (`load average 5.89` at 13:27:50; `state.db` checkpointing a **40 MB WAL** on the same
|
||
spindle). **Next lever is `state.db` checkpoint/retention discipline, not more caching.**
|
||
- **Cost of the fault:** *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and
|
||
**12:00:51 → 18:00:42 (5 h 59 m 51 s)** — its two slowest slots on record, both delivered; *FreshRSS Midday
|
||
Briefing* `ecc1fc13655a` took **2 h 09 m 53 s**; and the stocktake failed a third day. - **Cron ledger:** 11-08 settled **169 rows — 168 `completed` / 1 `failed` / 1 `running`** (this audit,
|
||
23:59:33). The single failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08**, on
|
||
a **new terminal signature**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still
|
||
pending and the model produced no follow-up text.` (the previous two red days read *"lost its durable fire
|
||
claim ownership"*). **`failure_streak: 3`.** *Working hypothesis, not proven:* a suppressed `pre_tool_call`
|
||
holding a tool result open is exactly what the error describes, but the string does not name the hook. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**. Fleet **29 jobs (26 enabled / 3 disabled)** — a
|
||
**sixth** consecutive audit at the same count, nothing added, retired, disabled or repinned (fleet
|
||
`updated_at` moved to `2026-11-09T00:06:50+08:00`, read as a run-state stamp only). - **Incidents:** unclosed rows **2 → 3** — a third silent `detected` row for the stocktake
|
||
(`1a8e90_a9cad99cb150`, `alerted_at: null`), all three still unreconciled. **The alerting path is not at
|
||
fault:** the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and closed `resolved` at
|
||
**08:06:54** (its admission-read `database is locked`, cleared when the run completed). - ✅ **Verified resolution 1 — the Mnemosyne Health Monitor's no-agent conversion works.** The 18:00 slot
|
||
(agent mode) ran **12:00:51 → 18:00:42 = 5 h 59 m 51 s**; the converted job's first slot ran
|
||
**2026-11-09 00:00:48 → 00:02:02 = 73 s**, exit 0, silent, writing a **159 B** output file against
|
||
**105,982 B** for the last agent-mode run. Coverage live at **working 1030/1030 = 100.0 %**, **episodic
|
||
1192/1192 = 100.0 %**, so the `< 95 %` alert is correctly quiet. The script compares with `awk` because
|
||
**`bc` is not installed on this host**. - ✅ **Verified resolution 2 — `state.db` re-measured and one corruption remnant removed.** Now
|
||
**1,890,656,256 B (1.89 GB)**, **2,901 sessions / 197,682 messages**, live **40,157,672 B** WAL. `state.db.corrupted.20260811_151052` (**624.1 MB** + 32 KB `-shm` + 0 B `-wal`) **deleted** in-window; `state.db.recovered.recovery.json` (14,566 B) remains and
|
||
`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (**23.6 MB**) was **not** cleaned. - **Pages updated:** `systems/conduit.md` (the pre-tool-call section, **total**), `systems/hermes-agent.md`
|
||
(hook budget + `state.db` size, **total**), `systems/scheduled-tasks.md` (new Count bullet, **total**),
|
||
`systems/mnemosyne.md` (health-monitor conversion verified, **total**), `current-state.md` (**total**),
|
||
`incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.md` (`state.db` 1.7 → 1.89 GB in three
|
||
places, re `confident`), `incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md` (open items),
|
||
`index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log. - **No wiki page removed or archived.** No system obsoleted; no decision reversed. - **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit. ### 2026-11-10 — Nightly audit (run 2026-11-09 23:59 AWST; window: 2026-11-09, live probes 00:0x–01:0x)
|
||
|
||
- **2026-11-09 is the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Deliveries **8/8 `delivered`**, 0 `for_failure`. Unclosed incident rows **3 → 0** (`cron_incidents`: 134 rows, 0 open). → `current-state.md`, [[systems/scheduled-tasks]]
|
||
- ⚠️ **The stocktake's `failure_streak` 3 → 0 / `last_status ok` are cosmetic.** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m**, closed `completed` (`suppressed`), and landed **zero edits and no commit** — its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09); it is still pinned to `custom:omni-101` / `free-stack`. New page: [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. The three long-open incident rows (`1a8e90_628d914f1f97`, `_2f370ef764f5`, `_a9cad99cb150`) all closed at **12:10:47.125205**, the instant the run ended — hot-loop alert closing, not a fix. - ⚠️ **New finding — Job Radar Stage 1 (11-09)** `dca8482e4f76` ran **07:00:48 → 16:10:12 = 9 h 09 m**, never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**), and both Stage-2 slots (`5fc292ff0f3b`) fired **before** it finished and declined silently. Same silent family as 11-02/11-03 with a new trigger (read loop). No incident row raised. New page: [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. - ✅ **11-08 `pre_tool_call` hook cascade did not recur** — no `pre_tool_call` timeout line in `errors.log`; only two benign skips; no 11-09 run outside the two long-runners exceeded 74 min. Follow-up appended to [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. - **Tailscale:** 58 non-funnel nodes (57 peers + `hermes`), 42 online. `octop` **online** again; 8 offline ages refreshed (cloudreve 108d, elementsynapse 111d, freshrss-new 84d, conduit 65d, octoagent 8d, omarchybook 19d, openmausbot 19d, ubuntu-server-xfce 81d); `codex-environment-*` last-seen stamps corrected. → [[infrastructure/tailscale]]
|
||
- **OmniRoute:** `omniroute-1` CT101 `/v1/combos` → HTTP 200, **55 combos**, `free-stack` present. `/v1/models` on both nodes and CT600 `/v1/combos` → **HTTP 401**, so the **6,480-model catalog figure (2026-10-26) stays unverified this window; no page changed.** → [[systems/omniroute]]
|
||
- **Not verified (gap recorded):** CT450/CT460 Hermes version strings (the `pct exec` probe resolves the stale root-level `/usr/local/bin/hermes`; CT460's `hermes` isn't on PATH in that shell) and their docker `Up` days (`lxc-attach … Failed to exec "docker"`) — [[infrastructure/hosts]] and [[infrastructure/docker-services]] version/uptime lines stand as written. Unchanged. - No secrets, keys, tokens or passwords written. - **Post-audit housekeeping (01:0x AWST).** Vault sync committed in `/home/hermes/wiki` (**`3a19147`**, both vault conventions; 20 files, then amended with the freshness report). `wiki-maintain.sh` re-run **after** the page-date corrections → **skills 299 / 67 categories** (`systems/skills-index.md`), freshness **2026-11-10: 0 stale / 127 fresh / 3 no-date of 132**, Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. Two long-stale pages re-verified and de-flagged: [[systems/push-notifications]] (bridge sha256 `59cabfcc…` unchanged; `vory-push` relay **live 01:00:34, embedded in another Hermes process** — a `pgrep` on the script name is *expected* to return nothing while healthy) and [[incidents/2026-10-18-degenerate-stream-canned-substitute]] (6 runs since 10-18, **0 failed**; newest 2026-11-09 12:00:41 `completed`). **Pushed:** `6de35a3` (maintenance) → this. ### 2026-11-11 — Stocktake: 36 stale pages re-verified; 0 stale; one materially stale fallback chain rewritten
|
||
|
||
- **Freshness:** the 03:00 `wiki-maintain.sh` run flagged **36 stale / 93 fresh / 3 no-date of 132** — the whole 2026-10-20 cohort (runbooks, concepts, decisions, comparisons, experiments, seven systems pages, the Gmail + Mnemosyne-Capture incidents, `CONTRIBUTING.md`, `agents/index.md`). After the corrections below, the **same-day report was regenerated** (script → fix → re-run) and reads **0 stale / 129 fresh / 3 no-date of 132**. Quartz rebuilt (807 files) and `hermes-wiki-serve.service` restarted. - **Skills:** `systems/skills-index.md` regenerates at **299 skills / 67 categories** — unchanged count from 2026-11-10; the only diff is the date bump. Verified the generator's arithmetic against the live tree by hand: **300 `SKILL.md` files** (excluding `.archive/`), **1 nested duplicate** (`productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` duplicating the same category+name) → **299 unique (category, skill) pairs**. The count is correct; do not "fix" it to 300 or 303. - ⚠️ **Materially stale page rewritten — [[runbooks/test-openrouter]].** The 2026-10-20 fallback chain was wrong in four of six slots. Live `hermes fallback list` (verbatim recorded on the page): primary **`smart-route` via `custom:omni-101`**, then `meituan/longcat-2.5-preview:free` (nous), `deepseek-v4.1-flash` (opencode-zen), `deepseek-v4.1-flash` (custom:guacano), `kilo-reflection-beam-free` (custom:litellm), `poolside/laguna-m.1:free` (kilocode), `or-inclusionai_ling-2_6-1t` (custom). `mimo-v2.5`, `stepfun/step-3.7-flash:free` and `qwen3.5:397b` via ollama-cloud have **left** the chain; two new entries **joined** at 3 and 4. `ollama.com` dropped out of the runbook's URL test list; `guacano`/`litellm` are noted as **internal + key-gated (401 without a key)**. → [[concepts/model-routing]], [[comparisons/model-routers]]
|
||
- ⚠️ **A false claim corrected — [[comparisons/hermes-vs-nanobot]].** The 2026-10-20 note claimed "CT333 is `zeroclaw` in the current Proxmox naming". Live `pct list` **and** `pct exec 333 -- hostname` both return **`clawtest`** — the wiki's original label was right; `zeroclaw` is only the daemon running inside it. The page now says so, dated. - ⚠️ **Incident signature change — [[incidents/2026-10-22-nous-portal-refresh-token-invalid]].** The refresh-token failure is **gone** (zero `invalid_grant` lines in `errors.log` through 2026-11-10; `auth.json` live, no `NOUS_API_KEY` in `.env`). What replaced it: **404 `insufficient_credits_for_paid_model`** for `anthropic/claude-haiku-4.5`, 3× on 2026-11-10. Cause identified: a **ninth auxiliary slot** `auxiliary.pulse_memory` points back at **`nous`** with a paid model — the 2026-10-25 remediation moved the original eight slots to `custom:omni-101` but this slot was added afterwards and is the only active nous path left. `nous` is still first in `fallback_providers`, but on a *free* model not observed failing. Open item revised from "re-authenticate" to "move `pulse_memory` or add credits". - **[[incidents/2026-08-29-gmail-read-state]] — the cron error state cleared, the root cause did not.** `goimapnotify` is **still active** (now PID 441), the Important-mail monitor (`d40085da631c`) is **still paused**, and `fetch_newsletters.py` **still calls `himalaya message read --preview`** on Anthony's INBOX — 74 days unfixed. The three email crons that read `error` on 2026-10-20 (`85edf6772bca`, `08eb86f02d37`, `06a0a1c8704f`) are now `ok` with `failure_streak: 0`, but green cron status is not evidence of safe behaviour: a job that marks mail read *and succeeds* still pollutes the mailbox. Both original mitigations are now inert. - **Live-vs-page diffs run (per skill pattern 12a), all claims held:** ct207 `cloudflared` ingress re-read in full (four Hermes entries unchanged, `mcp-portal` still absent — [[systems/cloudflare-tunnel]], [[infrastructure/domains-and-tunnels]]); CT205 Guanaco at `v0.8.15`/`5df435e` with Ollama 200 ([[systems/guanaco]]); CT270 `occ status` 33.0.2.2 installed/not-in-maintenance ([[systems/nextcloud]]); headless stack process/port state ([[systems/headless-server]], [[runbooks/restart-browser]]); hermes-ui 5174 on both CT460 and CT450 ([[systems/hermes-ui]]); all five `hermes-*` user units active + gateway/WebUI/Camofox effective cgroup limits ([[runbooks/restart-hermes]], [[runbooks/gateway-resource-resilience]]); `tool_loop_guardrails` and the whole `terminal:`/`tools:`/`memory.mnemosyne`/`custom_providers` blocks byte-identical; `ops` profile **still deleted 2026-10-11** while `multiplex_profiles: true` and both `profile_routes` still point at it — **31 days unresolved** ([[runbooks/multiplexer-setup]]); `/home/hermes/wiki` still a local-only repo, `/home/hermes/wiki-backups/` still absent, `~/.hermes/backups/` incident dirs intact; `m93p` **not** re-probed this pass (its 2026-10-20 guest list stands as written). - **Estate shrank again (probed, not inferred):** `qm list` on `pve` now holds only VMs **100, 101, 110, 282** (VM **310** gone; the wiki had listed it) and `pct list` holds 17 CTs with **CT 601 gone**. VM 403/VM 401 and the `open-claw` Tailscale peer remain absent on both hypervisors → [[agents/open-claw]], [[agents/index]], `CONTRIBUTING.md`. - **Delivery verified at the destination, not assumed:** before the commit, `http://100.118.5.51:9120/` and the changed pages (incl. `runbooks/test-openrouter`, `concepts/model-routing`, `incidents/2026-08-29-gmail-read-state`) all returned **200**, as did the external route `https://hermes.kangaroo-eel.ts.net/wiki/`. `CONTRIBUTING.md`'s access table was re-verified the same way and its `agents/open-claw/` scope row struck through ([[CONTRIBUTING]]). - **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page (the Gitea remote's embedded token is deliberately not reproduced). One pre-existing broken wikilink left as-is: `systems/guanaco.md` → `[[systems/ollama]]` (present in HEAD, page already qualifies it "if exists"). ## 2026-11-12 00:1x AWST - Nightly wiki audit (window 2026-11-10 23:59 -> 2026-11-11 23:59)
|
||
|
||
- **The audit's own failure is the headline.** The 2026-11-10 23:59 slot (`cbfe434a3b5e`) ran **12 m 35 s** (23:59:54.486572 -> 2026-11-11 00:12:29.646987) and died on the router's pre-dispatch filter: `HTTP 429 ... all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s` — **6 candidates, `attempted 0`**, tenant `tn-5ud77abtl1` quota exceeded, delivery `suppressed`. **Consequence: the 2026-11-10 window was never reviewed.** Recovery was implicit (pool reset ≈ **06:36:57**; the 07:0x cohort ran clean), and this audit deliberately covers **both** windows. New page: [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]; new index entry. - **The raised row was never alerted.** `cbfe43_ed01845310d8` (`rate_limit`) sits at `detected` with **`alerted_at: NULL`, `closed_at: NULL`** — the **only open row** in `cron_incidents` (the previous one, `5fc292_773822455856`, closed `resolved` at 08:03:47). Fleet non-clean `last_status` **1 -> 2** (`066bb5191280` Uber Eats `delivery_failed` streak 0, plus `cbfe434a3b5e` `error` streak 1). - **Both long-running incidents recovered, with output.** Stocktake `1a8e90ea2271` **03:00:56.081504 -> 04:11:38.724852 = 1 h 10 m 42 s**, `completed`, **36 stale pages re-verified -> 0 stale**, commits **`341dda9`** + **`c108eb1`** (freshness report `raw/stocktake/freshness-20261111.txt`, 6,235 B) — closing [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. Job Radar Stage 1 `dca8482e4f76` **07:01:08.358004 -> 07:37:34.076224 = 36 m 26 s**, writing the **first handoff since 2026-11-06** (`radar-ref/handoff/2026-11-11.json`, 22,682 B, mtime 07:59); Stage 2 `5fc292ff0f3b` recovery fire 07:35:56.632032 -> 08:03:47.816811 `delivered`, then 11:30:56 -> 11:35:18 and 14:30:55 -> 14:34:56 `delivered`, auto-closing the 11-10 row `resolved` — closing [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. Both incident pages set to `status: resolved` with a dated Recovery section. - **Ledger, both days re-parsed (times stored as ISO TEXT, grouped by `substr(started_at,1,10)`).** 2026-11-09 **174 completed / 0 failed**; **2026-11-10: 157 rows — 155 `completed` / 2 `failed`** (Radar S2 scheduled 11:30:35.267155 -> 22:18:49.814215 = **10 h 48 m 14 s** `interrupted` with `for_failure: 1`; the audit's own 12 m 35 s 429); **2026-11-11: 172 rows — 171 `completed` / 1 `running` / 0 `failed`**, 15 distinct jobs, **first zero-failure day since 11-09**. Longest 11-11 run **1 h 10 m 42 s** — **nothing over 71 minutes**, against 9 h 09 m (11-09) and 10 h 48 m (11-10). Fleet **29 (26 enabled / 3 disabled)** for a sixth window. - **Deliveries checked at the destination:** 2026-11-11 = **7 rows, 7 / 7 `delivered`, 0 `for_failure`** (07:11:42, 07:31:01, 08:03:47, 08:34:36, 12:19:38, 14:34:55, 18:45:58); 2026-11-10's late row (22:18:48) is the one `for_failure: 1`. 2026-11-12 so far: 5 rows, 4 `completed` / 1 `running`. - **Providers unchanged; symptoms worse.** `config.yaml` mtime still **2026-11-09 22:58:07** — nothing repaired this window. The expired `opencode-zen` key at `fallback_providers[1]` produced **7 x HTTP 401 `api_key_expired`** on 11-11 (07:01:13 -> 19:19:39), each **immediately after** nous returned **14 x 404 `insufficient_credits_for_paid_model`** (07:01:11 -> last 19:19:38, all from the still-open desktop session `20261031_190813_95fbde`). **Two consecutive dead hops on one chain** — [[systems/opencode-go]] updated from "latent risk" to observed failure; [[incidents/2026-10-22-nous-portal-refresh-token-invalid]] gained the 11-11 escalation block. `auxiliary.pulse_memory` remains the **only slot with no `fallback_chain`** and logged **19 lines / 11 refusals** on 11-11. - **Pages written:** 1 new incident; 2 incidents -> `resolved`; 2 system pages + `current-state.md` + `index.md` updated; `systems/scheduled-tasks.md` gained the 2026-11-11 count/state normalisation. `~/.hermes/logs/errors.log` is now **6,179 lines** through **2026-11-12 00:10:11**. **No page deleted, no decision reversed, no secrets written** — no credential value appears in any page; the opencode key is recorded by *state*, never by value. - **Post-audit housekeeping (00:2x AWST 2026-11-12), vault sync.** Touched files copied to **both** vault conventions — `…/agent-estate-wiki/<rel>` and `…/agent-estate-wiki/wiki/<rel>` — with `sha256sum` confirmed equal across all three copies; vault committed locally in `/home/hermes/wiki` (`master`, **no remote — never pushed**), explicit paths only, so the unrelated `reports/uber-eats/*` dirt stayed out. Before copying, each vault file was compared against the **pre-edit** authoring blob (`c108eb1`): every difference was a *stale-but-clean* older version (vault `index.md`/`skills-index.md` still 11-10/11-09, `log.md` still missing the 11-11 stocktake entry, the nous incident page still 2026-10-26 at 48 lines vs 96) — **no direct-to-vault edits to preserve**, so forward-sync was safe. The 2026-10-22 nous incident page was **created** in vault B's `incidents/` (it had never been mirrored); the new 11-10 audit page went into **both** `incidents/` dirs. - **Vault drift re-measured (hash sweep over all 133 authoring pages, this run's 10 touched files excluded):** **Vault A — 13 stale / 56 missing / 64 in sync. Vault B — 58 stale / 8 missing / 67 in sync.** Unchanged in character from the 2026-10-19 measurement (the copy bridge still only syncs what an audit touches), so the missing-page backlog is recorded here and **left to an explicit request** rather than silently swept — a 133-page mirror pass is bigger than a nightly audit's mandate. ⚠ This is now a two-night-old gap: because the 2026-11-10 audit died at 00:12, **nothing was mirrored for that window either.**
|
||
- **Freshness report:** `raw/stocktake/freshness-20261112.txt` (this run, 6,325 B) — **1 stale / 129 fresh / 3 no-date of 133** pages (was 132; **+1 = the new incident page**). The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding. `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. Quartz rebuilt and `hermes-wiki-serve.service` restarted; the new page returns **200** at the served site with its body content present. ## 2026-11-12 03:0x–04:1x AWST — Stocktake (03:00): tailscale drift corrected; 1 stale + 5 threshold pages re-verified live
|
||
|
||
- **Freshness:** `wiki-maintain.sh` → **1 stale / 129 fresh / 3 no-date of 133** (was 132, +1 = the new 2026-11-10 incident page). The single stale page was **[[incidents/2026-09-02-sftpgo-outage]] at 22d** — re-verified live and cleared. The 3 no-date pages are unchanged (`SCHEMA.md`, `experiments/active/example.md`, `log.md`). `systems/skills-index.md` regenerates identically at **299 skills / 67 categories**. - **⚠ Live-vs-page diff found real drift in [[infrastructure/tailscale.md]] (2d old — age scan would have missed it):** three peers read `Online: false` in the peer record while **all three ping fine** — `mailflow` (CT1262 m93p, pong `192.168.178.142:41641` 2 ms), `qwenpaw` (CT465 pve, pong `192.168.178.186:41641` 7 ms) and `springroll` (pong via **DERP(sin)** 210 ms, relay not direct) — with both LXCs `running` in `pct list`. All three rows stay **online**; the peer record is stale (the m93p pattern again). One **genuine** correction in the other direction: `omarchymac`'s row claimed `offline 2d / last seen 2026-11-08 00:45Z` when the live record reads `LastSeen 2026-11-07T06:41:48Z` = **offline 4d** and ping → **timeout**. 9 named offline ages + the six `codex-environment-*` ephemerals refreshed (+2d). No IP changes, no missing rows, no new peers (58 rows = 57 live IPv4 peers + `hermes`; 80 peers total / 57 IPv4 / 23 IPv6-only, unchanged). Table rebuilt by script, not by hand. - **Re-probed and re-confirmed:** CT299 `sftpgo` **running**, peer `Online: true`, `pong via 192.168.178.92:41641 in 0s`, healthz **401** (auth-gated, service up) — closing [[incidents/2026-09-02-sftpgo-outage]] as still-resolved, and noting that `pct config 299` now reads **`swap: 512`** (the 43 Gi over-provisioning in that root cause is gone). Both DEAD Serve targets (`8769` `/agent-estate`, `9123` `/synergy-prep`) still have **no listener** — re-confirmed a fifth time. CT201's Serve table re-probed unchanged (`/`→8085, `/mgr`→8084 on both `:443` and `:8443`). Container counts re-confirmed on **both** hypervisors incl. `qm list`: `pve` **17 LXCs + 4 VMs**, `m93p` **20 LXCs + 2 VMs**. **New guest found: CT1007 `homebridge` on m93p — running but `tailscaled` `inactive`**, so it runs without joining the tailnet (same class as CT105 conduit, CT1264 openmausbot). - **Incident page re-opened for a **recurrence** — [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]] (21d, tomorrow's stale list).** The *"all targets were skipped by pre-dispatch filters"* signature has now fired on a **second job and a second pin**: `cbfe434a3b5e` (Wiki Audit) failed on **2026-11-10** with `RuntimeError: HTTP 429: … pre-dispatch filters … (credits 100%)`, `failure_type: rate_limit`, and its incident row `cbfe43_ed01845310d8` closed `resolved` **2026-11-12T00:25**. The original subject has **not** repeated — `dca8482e4f76` ran clean on 11-06/09/10/11. Caveat recorded on the page: `executions.db`'s retention floor is **2026-11-06T05:09:30+08:00**, so the 2026-10-21 row is *not* re-readable from the DB — the new read is in-window only. - **Four threshold/near-threshold pages re-verified against live state (skill pattern 22), each with a dated note — no date bump without a probe:**
|
||
- [[incidents/2026-07-29-mnemosyne-relay]] (19d) — `tools: None` absent from `config.yaml`; **0** occurrences of `Unknown Mnemosyne tool` across 4 rotated gateway logs and **0** of the relay `scope handle` error; `mnemosyne stats` → 2782 memories and recall returns ranked results. - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] (18d) — `himalaya envelope list -a work -s 1` still fails on `login not supported`; no OAuth2 token in the himalaya config, so path 1 (Anthony's sign-in) remains the only route. - [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]] (19d) — keyed probes on `omniroute-101`: **`good-free` → HTTP 200 (`glm-5.2-cheap:free`)** and **`free-stack` → HTTP 200 (`stepfun-ai/step-5-preview`)**. The mitigation holds and neither combo now resolves to deepseek. **The three free-DeepSeek upstreams themselves were NOT probed**, so "upstream still degraded" is neither confirmed nor refuted and restoring deepseek-first ordering is still unjustified — recorded as such on the page. - [[reports/communities-wa-social-media-audit-2026]] (18d) — **not re-verified this pass** (live Department of Communities pages not re-fetched); its `review_after: 2026-11-26` still stands and its own note already scopes the unverified claims. Flagged, left as written. - **Order of operations:** script → page corrections → **re-run script** → log/index → commit. `wiki-maintain.sh` re-run after every page edit; Quartz rebuilt and `hermes-wiki-serve.service` restarted. - **No page deleted, no decision reversed, no secrets written.** No credential value appears in any page; the OmniRoute probe used the `key_env` name, never the value. - **Correction to the entry above (2026-11-12 00:1x AWST audit), verified live rather than inherited:** that entry states *"The single stale entry is **`log.md` itself**, which carries no frontmatter `updated:` — expected, not a finding."* **That is wrong.** The committed `raw/stocktake/freshness-20261112.txt` from that run (sha256 `da5f9e2e…`, byte-identical to this run's pre-fix output) reads **`- [ 22d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-10-21)`** as its single `## STALE (1)` line — the real page, which this stocktake has now re-verified and cleared. `log.md` sits in the `## NO updated DATE` bucket, as always (pattern 13). The stale page was therefore **left unflagged in prose for one cycle while sitting stale in the tree** — the exact "fresh-looking prose, stale content" failure the report exists to prevent. ## 2026-11-12 23:5x – 2026-11-13 00:4x AWST — Nightly audit (window 2026-11-11 23:59 → 2026-11-12 23:59)
|
||
|
||
**Clean day, incident-clean estate — and an upgrade the security scan would not allow.**
|
||
|
||
- **The estate's last open incident closed itself.** `cbfe43_ed01845310d8` (the 2026-11-10 audit's 429)
|
||
went `closed_at: 2026-11-12T00:25:36.133387+08:00`, `state: resolved`, when the **next slot of the same
|
||
job ran clean** — `cbfe434a3b5e` 2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = **25 m 42 s**,
|
||
`completed`, delivery carrying both windows. **`cron_incidents`: 0 unclosed of 137.** No config produced
|
||
it; the pool reset at ≈ 06:36:57 on 11-11 did. The 429 page's `status: open` was **stale** — corrected to
|
||
`resolved` with a new *Resolution* section, and its "only open row" wording superseded. → [[incidents/2026-11-10-wiki-audit-429-all-targets-skipped]]
|
||
- **Ledger, live from `executions.db`:** 2026-11-11 **172 / 172 `completed` / 0 `failed`**; 2026-11-12
|
||
**176 rows — 175 `completed` / 0 `failed` / 1 `running`** (this audit, 23:59:54); 2026-11-13 so far
|
||
7 / 7. **Second consecutive zero-`failed` day.** Deliveries **11-11: 7 / 7; 11-12: 9 / 9 `delivered`**,
|
||
0 `for_failure`. Fleet **29 (26 / 3)** — seventh window unchanged. Retention floor `2026-11-07T00:00:42`,
|
||
1,001 rows. - **NEW PAGE — [[incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan]].** On 2026-11-12
|
||
(~17:38–18:29) `hermes plugins install … --enable --force` for `hermes-hq-push` was refused:
|
||
*"Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec))"*, *"--force does not override a
|
||
dangerous verdict."* **Verified in source rather than assumed:** the block is one unconditional branch
|
||
(`tools/plugin_guard.py:406-418`); `--force` lifts `caution` only; **there is no per-verdict override,
|
||
env var, plugin trusted-source list or allowlist** — the only key is `plugins.scan_on_install` (default
|
||
`true`, currently unset), and setting it `false` disables scanning for *every* future install. The
|
||
`trust_level` machinery exists in `tools/skills_guard.py` (skills path) and is **not read** by
|
||
`plugin_guard.py:398`, which hardcodes `trust_level="community"`. Installed **v0.9.42 untouched,
|
||
`enabled`, still delivering**. **Fix = PR #2** on `mrcharlesiv/hermes-hq-plugins` (author `Tony0410`,
|
||
`fix/tile-skill-heredoc`, **OPEN / MERGEABLE**, `mergedAt: null`, +7/−2 in one file) — pushed from a fork
|
||
because `gh` here is authenticated as `Tony0410` with **read-only** access upstream; local clone
|
||
`/home/hermes/workspace/hq-plugins-fix` (`90c9d28`). - **`systems/hermes-hq-push.md` corrected against live probes:** version **0.9.27 → 0.9.42** in the
|
||
*Verified state* table (which still carried the 2026-11-04 reading), tree mtime **2026-11-07 06:42 →
|
||
2026-11-08 02:19** (measured), the config claim's **line 7556 → 13226** (block opens line 13210 — noted
|
||
as drifting), plus a new *2026-11-12 — refused upgrade* section and two new *Open* items. - **The "stored hold-and-ask instruction" was never stored.** Recursive sweep of `~/.hermes`
|
||
(**184,716 files**; generated/venv/browser-profile/SQLite trees excluded) for `wait for my answer`,
|
||
`show me what it found`, `security scan flags`, `security-scan` → **zero matches in any editable text
|
||
file**; the sentence exists only in `state.db` **conversation history** (58 rows). **Nothing removed, no
|
||
policy reverted** — there was nothing persisted to revert. Walkthrough recorded as the new skill
|
||
`stored-instruction-search-and-removal`; installed skills **299 → 300** (`skills-index.md` regenerated
|
||
by `wiki-maintain.sh`). - **Config: no drift.** Live `~/.hermes/config.yaml` is **byte-identical to the 2026-11-12 18:14:52
|
||
snapshot** (565,433 B). The only delta since the **11-09** snapshot is the **model catalogue block**
|
||
(lines 1916-1941): 6 entries removed, 9 added. Nothing unrelated moved. - **Still open, unchanged and re-confirmed:** the *"all targets were skipped by pre-dispatch filters"*
|
||
signature recurred on a **second job and a second pin** (2026-11-12 03:00 stocktake), and every
|
||
model-calling job still rides `custom:omni-101` with no failover for the free-tier cohort
|
||
→ [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]. - **Pages touched:** `incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md` (new),
|
||
`incidents/2026-11-10-wiki-audit-429-all-targets-skipped.md`, `systems/hermes-hq-push.md`,
|
||
`systems/scheduled-tasks.md`, `current-state.md`, `index.md`, `log.md`, `systems/skills-index.md`
|
||
(regenerated). **No page deleted, no decision reversed, no secret written.**
|
||
|
||
## 2026-11-13 23:5x – 2026-11-14 00:2x AWST — Nightly audit (window 2026-11-12 23:59 → 2026-11-13 23:59)
|
||
|
||
**Headline: a third consecutive `failed`-free day and a fully clean fleet — then the `pre_tool_call`
|
||
hook timeout came back and reopened its incident.**
|
||
|
||
- **Ledger — 2026-11-13: 174 rows, 173 `completed` / 0 `failed` / 1 `running`** (this audit,
|
||
`cbfe434a3b5e`, started 23:59:41). Trend: 11-11 **172 / 172**, 11-12 **176 / 176**, 11-13 **174 / 173**
|
||
— the **third consecutive zero-`failed` day**. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`**
|
||
(11-12: 9 / 9). *cron_incidents* **0 unclosed**. - **Fleet 29 (26 enabled / 3 disabled)** — an **eighth** consecutive window at the same count, nothing
|
||
added, retired, disabled or repinned. **Every job now reads `last_status: ok` with `failure_streak` 0:
|
||
the fleet has no non-clean row at all.** The last one cleared because **`066bb5191280` (Uber Eats Monthly
|
||
Spend Report) was fixed end-to-end on 2026-11-13** — so [[systems/scheduled-tasks]]'s *"the fleet's only
|
||
non-clean `last_status`"* is now a historical statement, not a current one. - **Uber Eats fix, recorded for the audit trail.** Two stacked root causes: (1) the job's **runtime prompt
|
||
had grown to 88 k chars** (it carried `camofox-browser-automation` + `browser-backend-verification`), so
|
||
the 2026-11-01 run exhausted its turn budget mid-format and shipped a truncated digest (*"remaining 4 in
|
||
the report"*) plus forbidden run commentary; (2) the `delivery_failed` label was a **stale
|
||
`TELEGRAM_HOME_CHANNEL_THREAD_ID=34795` env var** that only clears on a gateway restart — which happened
|
||
**2026-11-13 13:14**. Fix: a deterministic digest printer
|
||
**`~/.hermes/cron/scripts/uber-eats-digest.py`**, a rewritten prompt with a hard output contract, and the
|
||
skills cleared (**88 k → 1.8 k chars**); the lesson went into the `cron-job-maintenance` skill. Verified on
|
||
the **18:51 → 18:58** slot — **154 unique / 153 completed** orders, Oct **9 orders / $301.04** (unchanged),
|
||
delivery row `delivered`, `hermes cron doctor` clean across all 26 enabled jobs. Pages already regenerated
|
||
in-window by that work: [[reports/uber-eats/index]], `reports/uber-eats/2026-10.md` + `.html`
|
||
(commits `bc35c4b`, `a0ec3a1`). **This audit updated that job's row in [[systems/scheduled-tasks]].**
|
||
- ⚠ **[[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] REOPENED — `resolved` → `active`.** The
|
||
hook overran its **raised 120 s** budget at **2026-11-14 00:05:40**, followed by **10** *"skipped after
|
||
previous timeout or while still running"* lines through **00:06:38**. It refused **this audit's own** first
|
||
tool calls (first-hand: `skill_view` and `terminal` both returned the plugin timeout) for ~1 minute before
|
||
dispatch recovered unaided. `errors.log` single-source counts: **11-12 0 / 0, 11-13 0 / 0, 11-14 1 / 10** —
|
||
the **first `pre_tool_call` timeout in the retained log since the 2026-11-08 cascade**. **No cron run
|
||
failed.** ⚠ **The 2026-11-08 lines have rotated out of `errors.log`** (which now starts **2026-11-09
|
||
13:18:39**), so the page's 58-timeout figure **stands on the 2026-11-09 read and was not re-verifiable**
|
||
this window — recorded on the page rather than silently recopied. - ⚠ **Load is the trigger and it has roughly doubled.** `load average` **12.08 / 13.02 / 12.33** on **2 vCPU**
|
||
(2026-11-08 read: **5.89**). **`state.db` is now 2.04 GB** (**2,036,908,032 B**; **2,977 sessions /
|
||
207,577 messages** — was **1.89 GB / 2,901 / 197,682** on 11-09), and the root filesystem is **88 % used**. The **30 → 120 s fix is proven insufficient**, which settles the page's open hypothesis in the confirmatory
|
||
direction: at load ~12 the callback needs >120 s, so **there is no headroom left in the timeout**, and
|
||
re-raising it would only mask the fault. **`state.db` checkpoint/retention is now the estate's most urgent
|
||
open item.**
|
||
- **Freshness (2026-11-13 03:00 stocktake; file mtime 03:06:30): 0 stale / 131 fresh / 3 no-date of 134
|
||
pages.** Nothing needed retiring. → `raw/stocktake/freshness-20261113.txt`
|
||
- **Stocktake 2026-11-13 recovered to a normal window:** `1a8e90ea2271` ran **03:01:07 → 03:17:14 =
|
||
16 m 07 s**, `completed`, `last_status ok`, `failure_streak 0` (11-12: 22 m 10 s; the 11-08 failure was
|
||
1 h 34 m). It found **0 stale pages**, so there was nothing to re-verify and it **left no commit of its own
|
||
and no section in this log** — its freshness output was byte-identical to the copy already committed by
|
||
`e033a08`. Recorded as a **minor process gap, not a failure**: unlike the 2026-11-09 *"completes with zero
|
||
edits"* incident, there was genuinely nothing in scope. - **`executions.db` retention floor moved to 2026-11-08T00:01:20+08:00** (was 2026-11-06T05:09:30), so the
|
||
11-07/11-08 cascade **runs** remain readable even though their log lines have rotated. - **Pages touched:** `incidents/2026-11-08-pre-tool-call-hook-timeout-cascade.md` (frontmatter `resolved` →
|
||
`active`, plus a **Recurrence (2026-11-14)** section), `systems/scheduled-tasks.md` (new *Current as of
|
||
2026-11-13* section, the Uber Eats job row, frontmatter), `current-state.md` (new *Recent Changes
|
||
(2026-11-13)* section), `index.md` (new blurb, the incident's RESOLVED → REOPENED label, frontmatter),
|
||
`log.md` (this entry). **No page deleted, no decision reversed, no secret written.**
|
||
|
||
## 2026-11-14 (nightly audit; window 2026-11-13 23:00 → 2026-11-14 23:59 AWST)
|
||
|
||
**New:** [[systems/phonon-tts]] — the third TTS stack on CT200: `ptts-openai-server` (`:8881`, OpenAI-compatible `/v1/audio/speech`, model `gradium/phonon-40m-en`), `phonon-demo` (`:8882`, WASM demo UI on `nginx:alpine`), `ptts-ws-server` (`:8883`, WebSocket, locally-built `ptts-ws-server:0.7.1`). Verified live from CT460 — `/health` 200, `/v1/models`, and a POST to `/v1/audio/speech` returning a valid MP3; `:8883` 404s on a plain GET because it is a WebSocket endpoint. `/opt/ptts-voices` holds 7 voices. Page carries the container/timeline table and flags the **provenance gap after 09:03 AWST** — no retained session in this profile covers the 10:31 / 11:00 / 11:43 / 17:54 changes. **New:** [[incidents/2026-11-14-mnemosyne-capture-stall]] — **open incident**: Mnemosyne has captured nothing since **2026-11-14 09:32:59 AWST** (`working_memory` and `episodic_memory` stop in the same minute; last 8 working writes all 01:01–01:32 UTC). The health monitor `ea79a2107e36` escalated at **2026-11-15 00:03:35** (exit 1, incident `ea79a2_601037516abd`, `failure_type: auth`, state `alerted`, unclosed) *(Correction 2026-11-15 01:2x: this entry originally claimed ~12 h of unalarmed lead time, with the same hole logged ℹ️ informational at 12:00 AWST — **WITHDRAWN**. The 12:00:52 run measured `capture delay 29.4 min` (healthy, exit 0, suppressed) and saw nothing; the `-589.5 min` line belongs to the 00:03:35 run. The gate is activity-aware and magnitude-based. See the corrections pass below.)* ~15 h of desktop/Telegram/cron context uncaptured; the 23:30 sleep cycle consolidated **0 items**. Ruled out: the known stringified-config killer (`config.yaml` mtime 2026-11-13 18:03; YAML walk clean). Root cause **not established**. **Updated:** [[infrastructure/docker-services]] — CT200 re-verified 2026-11-15: **65 running / 72 total** (was 61 / 68 on 11-07), the three TTS containers added to the inventory, **no removals**. **Updated:** [[current-state]] and [[index]] — dated change log. **Ledger:** 2026-11-14 settled **167 rows — 166 `completed` / 0 `failed` / 1 `running`** (the running row is this audit, started 23:59:37); deliveries **4 / 4 `delivered`**; **`cron_incidents` 0 → 1 unclosed** (the monitor row above, `alerted_at` 00:03:44). 2026-11-15 so far: **8 rows — 7 `completed` / 1 `failed`** (the monitor), 1 delivery `delivered`. Fleet **29 jobs (26 enabled / 3 disabled)** — **non-clean `last_status` 0 → 1** (`ea79a2107e36`, `error`, `failure_streak` 1), the first red row since the Uber Eats job cleared on 2026-11-13. **Not changed:** `errors.log` for 11-14 carries only the benign recurring `Browser Use viewing unavailable` plugin warning (~2/hour) plus the already-recorded `pre_tool_call timed out after 120s` at 00:05:40 — see [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] (no new occurrence in the 11-14 body). ## 2026-11-15 00:5x — corrections pass on the 2026-11-14 entry (same audit window)
|
||
|
||
A second read of the live DBs tightened three claims made in the entry above. All three are applied to the pages; this entry records the corrections rather than leaving two versions in the log. 1. **WITHDRAWN — "one net-added CT200 container unidentified."** The `61 → 65` comparison mixed two different counts: the 2026-11-07 row's **61 was an explicit *non-seeder*** figure while today's **65 was a raw `running`** figure that **includes** the transient `worldmonitor-seeder-scheduled`. A set-wise name diff settles it: live = documented **+ exactly 3** (`phonon-demo`, `ptts-openai-server`, `ptts-ws-server`), **zero removals**, and **64 non-seeder running** against 61 on 11-07. **No unexplained container exists.** [[infrastructure/docker-services]] corrected. 2. **`cron_incidents` went 0 → 1 unclosed** — not "0 unclosed before the 00:03 alert". **138 rows** total; the open one is `ea79a2_601037516abd` (`state: alerted`, `alerted_at` 00:03:44.5, `failure_type: auth` — mislabelled). The monitor's **own** job is also the fleet's **single non-clean `last_status`** (`error`, `failure_streak` 1) — the **first red fleet row since 11-13**. [[systems/scheduled-tasks]], [[current-state]] and [[index]] corrected. 3. **RESOLVED (was listed as unverified) — did the 00:03 escalation actually deliver?** Yes. `deliveries.db` holds exactly one 2026-11-15 row: **`2026-11-15T00:03:39.052888+08:00`, `status: delivered`, `for_failure: 1`** — **3 s after** the incident row was raised, **5 s before** `alerted_at`. The **notification path worked; the detection path did not.**
|
||
|
||
Ledger precision as read at 00:5x: **2026-11-14: 167 rows — 166 `completed` / 0 `failed` / 1 `running`** (this audit, `cbfe434a3b5e`, 23:59:37); **2026-11-15 so far: 8 rows — 7 `completed` / 1 `failed`** (the monitor); deliveries 11-14 **4 / 4 `delivered`** (07:06 / 07:30 / 08:32 / 12:05). Pages also updated in this pass: [[incidents/2026-11-14-mnemosyne-capture-stall]] (delivery question closed, fleet line corrected), [[systems/mnemosyne]] (new ⚠ OPEN block at the head of *Known Issues*, frontmatter to 2026-11-15, and the naive-UTC trap note extended with the monitor's own raw-UTC alert text as a second occurrence of the same trap), [[systems/scheduled-tasks]] (new `### Open 2026-11-15` block under *Known Issues* + the monitor row in the disabled table annotated — **the threshold tests the *sign* of the capture delay, not its magnitude, which is why 12 h of a growing hole raised nothing**). ## 2026-11-15 01:0x — vault mirror sync + first drift measurement (same audit window)
|
||
|
||
**Vault:** `/home/hermes/wiki` (its own git root; local-only — `git remote -v` empty, so commit, no push). Touched files were mirrored into **both** vault conventions (`agent-estate-wiki/<rel>` and `agent-estate-wiki/wiki/<rel>`): **14 writes, sha256 verified per file, all OK**; vault commit **`b7d7f18`**, tree clean afterwards. Note the previous run's claim that `/home/hermes/wiki` "does not exist" was **wrong** — the directory and its repo are present and were last synced by the 11-14 run (`b067b97`). **Pre-check first** (skill §8 / §12b: compare each existing vault copy against the **pre-edit** authoring blob, `16767ef`, never the current one): 12 of 14 targets were `stale-but-clean` (identical to that pre-edit blob) or genuinely new. **2 conflicts investigated rather than clobbered** — `infrastructure/docker-services.md` in **both** conventions was dated **2026-11-06**, i.e. older than the pre-edit blob (11-07), with an 18-line diff that was entirely removals/date regressions and no unique lines → proven stale, forward-synced. Reading: **no audit between 11-07 and 11-14 ever mirrored that page.**
|
||
|
||
**Drift measured** (authoring HEAD vs vault, 136 tracked `*.md`), after this run's sync:
|
||
|
||
| convention | match | older-revision | no-authoring-match | missing |
|
||
|---|---|---|---|---|
|
||
| `agent-estate-wiki/` | 68 | 12 | **0** | 52 |
|
||
| `agent-estate-wiki/wiki/` | 68 | 56 | **1** | 7 |
|
||
|
||
Method: each "differing" copy was matched against the last **25 authoring revisions of its own path** — a match to an older revision is *stale-but-clean*, no match is an *orphan*. Every differing copy in the top-level convention is an older revision. **The missing/52 and missing/7 columns are the standing drift and are unchanged by this run** (skill §8: a full mirror pass is bigger than an audit's mandate — recorded, not attempted). ⚠ **One orphan, and it is the reverse of the usual case — the vault holds content the authoring tree has never had.**
|
||
`agent-estate-wiki/wiki/incidents/2026-10-31-conduit-voice-local-mode-not-applying.md` is **238 lines** with the root cause evidenced (`/v1/pool`, controlled realtime sessions against the Mac socket, title "…**rejects `session.update` in every form**"). The authoring page has been **103 lines at every commit that ever touched it** (`0c826f2` 2026-11-01 00:30, `a0b3734` 04:48, HEAD), and `git log -S'/v1/pool'` finds **nothing** — that evidence never entered the authoring tree. Vault mtime **2026-11-01 02:56**, i.e. written **straight into the vault** ~2¼ h after the authoring commit and ~2 h before `a0b3734` ("fix the Quartz build (YAML title)") normalised the title. **Left untouched and not forward-synced** (§12b: investigate, never clobber). **This needs an owner decision:** either port the vault copy back into `/home/hermes/workspace/agent-estate-wiki` (keeping the Quartz-safe `title`) or declare the shorter authoring page canonical and record the vault copy as historical. Until then the two trees disagree about the Conduit local-voice-mode root cause. ## 2026-11-15 01:2x — corrections pass: the Mnemosyne escalation's mechanism (re-derived)
|
||
|
||
**Three claims published in the 2026-11-14 entries of this log are WITHDRAWN** — they were reasoned from a
|
||
mis-attributed log line rather than read from it. Corrected against `~/.hermes/logs/mnemosyne-health-monitor.log`
|
||
by line number (file = **1103 lines**):
|
||
|
||
- **WITHDRAWN: "the 12:00:52 AWST run saw the same hole and logged it ℹ️ informational (`capture delay -589.5 min`),
|
||
so ~12 h of lead time produced no alarm."** The `-589.5 min` line is at **line 1046**, i.e. *inside the 00:03:35
|
||
run's own block* (header line 1030; the ❌ is line 1047). The 12:00:52 run (header **line 957**) measured
|
||
`capture delay 29.4 min` at **line 973** — healthy, exit 0, delivery `suppressed`. The monitor did not misread
|
||
anything at 12:00. - **WITHDRAWN: "the job exited 0 at 12:00 and 18:03."** Schedule is `0 */12 * * *` — 2026-11-14's only slots are
|
||
**00:09:12** and **12:01:52**. *(Correction 2026-11-15 01:2x: the earlier "the job exited 0 at 12:00 and 18:03" sentence is **WITHDRAWN** — the schedule is `0 */12 * * *`; 2026-11-14's only slots were 00:09:12 and 12:01:52.)* - **WITHDRAWN: the "threshold tests the sign, not the magnitude" explanation.** `mnemosyne_freshness_gate.py` fires
|
||
on magnitude — `delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min. The earlier slots
|
||
were silent because the gate is **activity-aware**: it can only flag a message with no capture row, and every user
|
||
message up to the 09:03 AWST `ios` turn had one (final capture 09:32:59, +29.4 min later). The conversation was
|
||
then idle until the **19:22 AWST desktop** turn — the first uncaptured message — which the next slot (00:03:35)
|
||
flagged at `-589.5 min`. Detection latency after detectability: **4 h 41 m** — cadence, not a bug. **What genuinely went unread — the corrected lesson.** `consolidation_log` shows the sleep cycles firing every
|
||
~31 min and returning **0 items for 29 consecutive cycles** (streak opening `2026-11-14T02:04:42+00:00` =
|
||
**10:04:42 AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every monitor
|
||
run printed `Last consolidation: … (0 items)` — including the "healthy" one — but the gate checks consolidation
|
||
**age** (`> 36 h`) only, never the item count. **An empty-but-current cycle is the failure signature of a dead
|
||
write path, and nothing watches it.** Candidate fix: an N-consecutive-zero-item-cycle exception in
|
||
`mnemosyne_freshness_gate.py` (**not implemented — needs Anthony's approval**). ⚠ **Format trap, verified:** `consolidation_log.created_at` mixes formats — rows 4640–4651 carry `+00:00`, rows
|
||
4620–4622 are **naive local (AWST)**. Compared as one axis they misplace events by 8 h. ⚠ **`failure_type` is routing metadata, not diagnosis:** this job's twelve earlier `cron_incidents` rows read
|
||
`agent` ×3 / `timeout` ×1 / `unknown` ×8; estate-wide the table is 53 `unknown` / 47 `agent` / 15 `timeout` /
|
||
14 `script` / 5 `auth`. The job is `no_agent`, so `agent` is impossible — read `auth` as unexplained, not as a
|
||
credential fault. Pages corrected: [[incidents/2026-11-14-mnemosyne-capture-stall]] (false section rewritten, `auth` bullet upgraded
|
||
from suspicion to evidence), [[systems/scheduled-tasks]], [[systems/mnemosyne]], [[index.md]]. The incident itself
|
||
remains **OPEN / `alerted` / unclosed**; the *cause* of the stall is still **not established**.
|
||
## 2026-11-15 03:00 — stocktake (wiki-maintain.sh + drift re-verification)
|
||
|
||
Script run: skills-index regenerated (299 skills / 67 categories), freshness report
|
||
`raw/stocktake/freshness-20261115.txt` — **STALE (1)**:
|
||
`incidents/2026-10-09-cron-provider-timeout-storm.md` (22d, updated 2026-10-24). Pages ON the
|
||
threshold (21d): `incidents/2026-10-22-conduit-memory-start-timeout.md`,
|
||
`reports/communities-wa-social-media-audit-2026.md` (static deliverable, no live claims — left
|
||
unbumped). Pages at 20d (not re-probed this run, next candidates):
|
||
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md`,
|
||
`systems/current-profile.md`, `systems/memory-backend.md`, `systems/model-providers.md`,
|
||
`incidents/2026-10-03-ct450-hermes-update.md`,
|
||
`incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md`,
|
||
`incidents/2026-10-24-conduit-call-request-refused.md`,
|
||
`incidents/2026-10-25-gmail-digest-runaway-and-refire.md`.
|
||
|
||
**Re-verified live (storm page, 22d):** all 12 `cron_incidents` rows still `resolved`; weekly job
|
||
`1b23c042e17d` `last_status: ok` (2026-11-13). The one open fleet-wide incident
|
||
(`ea79a2_601037516abd`, first seen 2026-11-15 00:03) is the capture-stall monitor firing, not a
|
||
storm row. `executions.db` retention now starts 2026-11-09 — the page's 2026-10-09 execution
|
||
counts are no longer DB-re-derivable; noted on the page. Bumped with a dated verification note.
|
||
|
||
**Regression found (conduit page, 21d):** the Conduit voice memory-timeout fix was **lost in the
|
||
plugin's restructure**. `~/.hermes/plugins/conduit_push` is clean at upstream `0.168.2`; the
|
||
memory layer moved to `dashboard/api/memory.py` where `MEMORY_TIMEOUT_S = 10.0` is hard-coded
|
||
again (no `CONDUIT_MEMORY_TIMEOUT_S` env hook; restructure landed ~2026-10-29 AWST). Running
|
||
dashboard (since 2026-11-10) carries it. One real call since (2026-11-14 08:57 AWST) hit a warm
|
||
provider, so the cold-start path is untested. Page → `status: degraded` with a dated regression
|
||
section; **fix not re-applied** (needs Anthony's call: re-port + commit locally, or upstream).
|
||
Pages corrected: [[incidents/2026-10-09-cron-provider-timeout-storm]],
|
||
[[incidents/2026-10-22-conduit-memory-start-timeout]], [[index.md]].
|
||
|
||
|
||
## 2026-11-16 00:2x — nightly audit (window 2026-11-15 00:00 – 23:59 AWST)
|
||
|
||
**Nothing new was created and nothing was archived — the window's findings all land on pages that already existed. The memory incident deepened, and a second job failed.**
|
||
|
||
**Ledger — 2026-11-15: 168 rows, 165 `completed` / 3 `failed`** (2026-11-14 settled **167 / 167 / 0**, so nothing carried over). The three: `ea79a2107e36` (Mnemosyne Health Monitor) `00:00:53.928 -> 00:03:44.604`; `96eb95ce12f2` (Mnemosyne Watchdog) `11:31:20.699 -> 11:49:10.534610` (**17 m 49.8 s**, exit 1 — `runtime deps guard could not run: Command '['bash', '/home/hermes/.hermes/scripts/hermes-runtime-deps-guard.sh']' timed out after 600 seconds`, a **new signature** for this job; the `11:45` slot was **skipped** and `12:01:25.120952` completed, so it self-recovered with one slot lost); `ea79a2107e36` `12:01:24.719 -> 12:03:20.430`. Deliveries **9 / 9 `delivered`**, 3 `for_failure`. Fleet **29 (26 / 3)** — a **tenth** window unchanged; non-clean `last_status` still 1 (`ea79a2107e36`, `error`, streak **3**).
|
||
|
||
**Capture stall: still open at 38 h 46 m, and the freeze is total.** `working_memory` **1130** / `episodic_memory` **1249** — *identical at every read across the stall* (11-15 00:4x, 11-15 12:03, 11-16 00:18), so this is a hard stop, not degradation. Newest rows are still the `[MODEL_REFRESH_PROPOSAL]` maintenance writes at `09:32:58`/`09:32:59` AWST; newest **user-turn** capture `2026-11-14 09:04:19`.
|
||
|
||
**Decoded this run — the monitor's "≈4.64 h" is not the stall.** `mnemosyne_freshness_gate.py`'s `newest_user_message()` selects the newest `messages` row with `role='user'` whose session `source NOT IN ('cron','subagent')` (`NON_CONVERSATIONAL`, line 52) and reports **that message's** age — here the **Pulse** turns on `source: desktop`, ~6 h cadence. Every `0 */12` slot lands ~4.6 h after one, which is why all three alerts read alike: 11-15 00:03:35 → **11-14 19:22:30**; 11-15 12:03:14 → **11-15 07:23:25**; 11-16 00:03:15 → **11-15 19:22:59**. Proof capture is dead rather than idle: the 11-14 00:09:12 run logged `last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min`, and four 11-15 Pulse turns (`07:00:43`, `07:23:25`, `13:23:00`, `19:22:59`) now have no capture row. **The estate's only automatic alarm has therefore understated severity on every fire.**
|
||
|
||
**New mechanism recorded — incident-signature fragmentation.** `cron_incidents` **1 → 3 unclosed** (138 → **141** rows): `ea79a2_601037516abd`, `ea79a2_5f817708701e`, `ea79a2_75303493d484` — all this job, all `failure_type: auth`, all `state: alerted`, each `first_seen_at == last_seen_at`. The signature embeds the elapsed-hours float (`4.64157453` / `4.634034074444445` / `4.643481075555555`), which is never equal twice, so one durable fault mints a fresh unclosed row every slot and no row escalates. **Fix proposed, not applied** (normalise the value out of the signature) — it needs a code change on the monitor's incident path.
|
||
|
||
**Consolidator: alive, and every cycle is a deliberate no-op.** Latest row **4696** at `2026-11-15T16:14:14.934306+00:00` (= 00:14:14 AWST), `items_consolidated: 0`, `summary_preview` `No-op consolidation cycle — timestamp updated for watchdog` — the cycle **bumps its own timestamp to satisfy an age test**, so an age-only gate reads "fresh" forever. Streak now **~74 consecutive zero-item cycles** (opened `2026-11-14T02:04:42+00:00`). A **zero-item-streak** threshold is what would have caught this at 10:04 on 11-14.
|
||
|
||
**Sessions:** 9 started on 11-15, **all `source: cron`** — no interactive session, so no human-side change the wiki missed. Freshness (03:00 stocktake): **0 stale / 133 fresh / 3 no-date of 136**.
|
||
|
||
Pages updated: [[incidents/2026-11-14-mnemosyne-capture-stall]] (status re-read, the `4.64 h` decode, the three-row table + fragmentation mechanism, the watchdog slot loss), [[systems/mnemosyne]], [[systems/scheduled-tasks]], [[current-state.md]], [[index.md]]. Root cause of the stall: **still not established** — the incident stays `active`, `closed_at: null`.
|
||
## 2026-11-17 00:2x — nightly audit (window 2026-11-16 00:00 – 23:59 AWST)
|
||
|
||
**Verdict: 1 new page + 7 page updates, nothing archived, no new system.**
|
||
|
||
**Window (11-16):** quiet by every measure except the one already open. Ledger **175 rows — 173
|
||
`completed` / 2 `failed`** (both `ea79a2107e36`, the Mnemosyne Health Monitor, now `failure_streak: 5`);
|
||
deliveries **11/11 `delivered`**, 2 `for_failure`; **16 cron sessions, zero interactive**; the 11-15
|
||
watchdog deps-guard 600 s timeout did **not** recur; `state.db` 2.09 GB / WAL 40 MB / `/` 88 % / load
|
||
6.19–7.48. **No new system, no new concept, no page worth archiving.**
|
||
|
||
**Prior-window correction — the substantive finding.** While backing up 11-15 (the window the previous
|
||
audit covered), exact greps of `errors.log` turned up an unrecorded **plugin load-timeout storm** on
|
||
**2026-11-15**: **52** `Failed to load plugin '…': load timed out after 10s`, **519** discarded
|
||
`register_*()` calls — `conduit_push` **474** of them across eight load events, i.e. Conduit's tools,
|
||
skills and hooks were silently absent for those processes — **11** `pre_tool_call` timeouts (5 in
|
||
01:07–01:26, 6 in 12:18–12:49), **2** `pre_llm_call` timeouts, **10** refused tool calls. 11-14 had 4 / 0
|
||
/ 1 / 1 / 4; **11-16 and 11-17 are 0 across the board.** No fix was applied and the 10 s loader budget is
|
||
a code default (there is no plugin-load-timeout key in `config.yaml`, only `hook_callback_timeout: 120`).
|
||
|
||
**Pages:**
|
||
|
||
- **New** [[incidents/2026-11-15-plugin-load-timeout-storm]] — counts by day and by plugin, the
|
||
load-failure/registration-drop distinction, the eight `conduit_push` events, clustering, and the
|
||
open question (fail-open loader; a registration health check is proposed, **not applied**).
|
||
- [[incidents/2026-11-14-mnemosyne-capture-stall]] — 11-17 re-read: **62 h 54 m**, counts identical for
|
||
the fifth read, unclosed `alerted` **3 → 5** (143 total) with the two new incident ids, monitor streak
|
||
**5**, zero-item streak **120** (was 74); title raised 38+ → **62+ h**.
|
||
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] — **Recurrence 2 (11-15)**: 11 timeouts in
|
||
two clusters, 10 refusals, call-level blocks to 120.63 s, plus the note that a hook cluster can fire
|
||
**without** a loader failure (12:18–12:49). Load average **removed** as a claimed predictor.
|
||
- [[systems/conduit]] — 474 dropped registrations and what they cost.
|
||
- [[systems/mnemosyne]] — OPEN paragraph carries the 11-17 numbers.
|
||
- [[systems/scheduled-tasks]] — 11-17 Count bullet (fleet, ledger, deliveries, streak, the 11-16 clean-up).
|
||
- [[current-state]] and [[index]] — 11-16 window sections; `Updated:` / `Last updated:` lines bumped.
|
||
|
||
**Freshness:** `reports/communities-wa-social-media-audit-2026` is the only page outside threshold
|
||
(23 d). It is a static deliverable with `review_after` 2026-11-26 — **left unbumped deliberately**, and
|
||
recorded here so the next audit does not re-litigate it.
|
||
|
||
**Still open from earlier windows:** the capture stall has no established cause and no fix applied
|
||
(`status: active`, `review_after` 2026-11-18); the incident-signature fragmentation fix remains
|
||
**proposed, not applied**; the plugin-load/hook-budget health check is **proposed, not applied**.
|
||
|
||
## 2026-11-18 03:xx — 03:00 stocktake (cron)
|
||
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged vs 11-17), freshness report `raw/stocktake/freshness-20261118.txt` — 9 stale.
|
||
- Live-vs-page re-verification of stale pages; 7 re-verified with dated corrections:
|
||
- `runbooks/provider-health.md` — snippets re-executed: CT600 /models 200 (6,501), omni-101 /api/health 200, fallbacks 200.
|
||
- `runbooks/update-hermes-safely.md` — prerequisites re-verified (v0.23.0, venv + managed runtimes present).
|
||
- `incidents/2026-10-03-ct450-hermes-update.md` — CT450 now v0.23.1 (0 behind origin/main), CT460 v0.23.0; CT255 mattermost running + peer online → Mattermost-timeout claim superseded.
|
||
- `incidents/2026-10-03-mnemosyne-vector-recall-fts-only.md` — coverage monitor ok/streak 0; fix holds (script blocked by lifecycle guard; verified via monitor).
|
||
- `incidents/2026-10-25-gmail-digest-runaway-and-refire.md` — RESOLVED: last_status ok, one clean daily run 11-12→11-17, no refires; delivery_queued cleared.
|
||
- `incidents/2026-10-16-omniroute-503-resource-pressure.md` — 22 days clean, stays monitoring.
|
||
- `incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` — no unknown rows in retained window; 10-20 log line aged out (rotation noted on page).
|
||
- Left stale (flagged only, not verifiable this run): `reports/communities-wa-social-media-audit-2026.md` (24d), `incidents/2026-10-24-conduit-call-request-refused.md` (23d, open behavioural).
|
||
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
|
||
|
||
## 2026-11-19 03:xx — 03:00 stocktake (cron)
|
||
- Ran `wiki-maintain.sh`: skills-index regenerated (299 skills / 67 categories, content unchanged), freshness report `raw/stocktake/freshness-20261119.txt` — 3 stale.
|
||
- All 3 stale pages re-verified against live sources with dated corrections:
|
||
- `incidents/2026-10-24-conduit-call-request-refused.md` — `conduit-calls.json history` still 0 entries (call path unexercised ~26d); status stays open.
|
||
- `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` — both `ea79a2_*` incident rows closed/resolved (c3ab2d784ba3 10-27, a4661554a3c2 closed 11-07 15:43), error string absent from retained executions window → status → resolved; drain-policy follow-ups 1/3/4 remain open design questions.
|
||
- `reports/communities-wa-social-media-audit-2026.md` — live wa.gov.au check: remit verbatim-identical, five Ministers unchanged, site updated 18 Nov 2026 with fresh announcements.
|
||
- Quartz rebuilt + deployed via second `wiki-maintain.sh` run (pattern 13).
|