11 KiB
11 KiB
Anthony's Systems Wiki
Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-10-19
⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor,
m93p(100.104.51.98, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.
Start here
- current-state — Current systems status
- systems/hermes-agent — Hermes Agent overview and version
- systems/current-profile — Active Hermes profile details
Systems
Core
- systems/hermes-agent — Agent version, install, config paths
- systems/current-profile — Active profile details
- systems/nextcloud — Nextcloud file sharing (CT270)
- systems/guanaco — Guanaco LLM proxy on CT205 (Ollama stack)
- systems/model-providers — LiteLLM + OmniRoute providers, routing setup
- systems/memory-backend — Mnemosyne vector memory, settings
- systems/mnemosyne — Mnemosyne cron jobs and config evidence
- systems/omniroute — OmniRoute catalog, combos, and host path
- systems/openrouter — OpenRouter current state and history
- systems/opencode-go — OpenCode Go subscription, model limits, pricing
- systems/nanobot — Nanobot current state and history
- systems/browser-backend — Camofox (primary,
systemd --useron CT460, REST:9377) + Steel (disabled); memory-containment drop-in and the 2026-10-12 ceiling event documented - systems/hermes-ui — Browser Hermes Desktop (przbadu/hermes-ui)
- systems/paseo — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
- systems/ignis — Ignis browser-native Obsidian (wiki interface)
- systems/terminal-backend — Shell persistence, working directory
- systems/tool-search — Always-on lazy discovery for external tools
Integrations
- systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped (peer offline 29d; host answers neither
:8065nor ICMP). - systems/push-notifications — Cadu/Vory push bridge (
hermes-push,vory-push): gateway on:9119, state in~/.hermes/push/. Broken —connected: false(the bridge dials a name that resolves to127.0.1.1where nothing serves 443); chronic, not a blip. - systems/scheduled-tasks — 28 cron jobs (25 enabled, 3 disabled); 7 carry a non-clean
last_statusas of 2026-10-19 00:2x — down from 12. The 2026-10-18 window was clean except one run:ecc1fc13655a(FreshRSS Midday) delivered a canned non-answer to Telegram and then died on a model repetition loop → incidents/2026-10-18-degenerate-stream-canned-substitute. Theomniroute-1503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00) cost 9 model-call failures across 7 jobs and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See incidents/2026-10-16-omniroute-503-resource-pressure. Prior incident: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs and ~34 h of digests blanked. All 18 model-calling jobs still sit oncustom:omni-101with no failover. - systems/skills-index — Installed skills inventory (346 skills / 70 categories; regenerated daily by the 03:00 stocktake job). Note: 347
SKILL.mdfiles exist on disk — the generator de-dups one nested duplicate, see systems/scheduled-tasks. The 2026-10-17 slot failed (HTTP 503), but the 10-18 03:00 slot closed the hole:freshness-20261018.txtwritten, index regenerated and committed (dd53e5f) — freshness 47 stale / 41 fresh / 7 no-date. ⚠ The 2026-10-13 regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change:updated/verified_on10-12 → 10-13).
Incidents
- incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (
ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic:346at 12:08:11, then died at 12:08:25 onRuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? - incidents/2026-10-16-omniroute-503-resource-pressure — OmniRoute's second failure mode, 2026-10-16 → 10-17:
omniroute-1load-shed model calls withHTTP 503 … resource pressure— 9 model calls across 7 jobs over ~19 h, while/api/healthstayed200 okand/v1/modelsstayed401. Self-cleared (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; 8 open overall, and the ledger tags 7 of the 9failure_type: unknown. Single-node topology still open. - incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:
omniroute-1(100.78.63.20:20128, providercustom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved) - incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST; 11 of its 12 incidents are now closed (five on 2026-10-10, the last hold-outdca848_23fcd2b18aaaon 2026-10-12 07:33:41). 1 remainsalerted—1b23c042e17d, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
- incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (
cliprox/terra,qwen3.5:397b,deepseek-v4-flash,gpt-5.6-terra) repinned to routing aliases oncustom:omni-101; failing set 7 → 0 (resolved 2026-10-05) - incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
- incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
- incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute
cliprox/terraconnection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin - incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only
XOAuth2, himalaya configured for password (open since 2026-08-26) - incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing
dotenv/mnemosynebroke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env —mnemosyneandelevenlabsmissing from the committed uv env the gateway loads; both repaired and verified the same evening - incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
- incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
Monitoring & Dashboards
- systems/homepage-dashboard — Homepage dashboard (CT201)
- systems/home-assistant — Home Assistant integration
Reports
- reports/uber-eats/index — Uber Eats monthly spend reports
- reports/communities-wa-social-media-audit-2026 — Department of Communities WA social media audit and first-day brief
- systems/obsidian — Obsidian vault structure and conventions
- queries/retained-research-answers — Retained research answers (fastest VNC clients, 2026-10-11)
Infrastructure
Services
- infrastructure/hosts — Hosts, containers, important paths
- infrastructure/docker-services — Docker compose files
- infrastructure/credentials-map — Where credentials live (no values)
- infrastructure/network-map — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
- infrastructure/domains-and-tunnels — martinwa.org hostnames and Cloudflare Tunnel routing
- infrastructure/backups — hermes backup CLI, retention, and gaps
Networking
- infrastructure/proxmox — PVE hosts (two:
pve8.x +m93p9.2.0, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd088.17 % / ~63 GiB free, 2026-10-14) - infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 48-node tailnet inventory (re-verified 2026-10-19:
guest-vps,openmausbot,openmausbot-vmadded;mattermostoffline 29d) - systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly
Concepts
- concepts/agent-memory — Memory model
- concepts/model-routing — Routing concepts
- concepts/persistent-shells — Terminal persistence
- concepts/browser-automation — Browser automation concepts
- concepts/tool-calling — Tool calling patterns
Decisions
- decisions/index — Decision index
- decisions/keep-openrouter-as-fallback — Keep OpenRouter as fallback provider
- decisions/persistent-shell-enabled — Enable shell persistence
Comparisons
- comparisons/camoufox-vs-chrome — Camoufox vs Chrome
- comparisons/hermes-vs-nanobot — Hermes vs Nanobot
- comparisons/memory-backends — Memory backend comparison
- comparisons/model-routers — Model router comparison
Runbooks
- runbooks/multiplexer-setup — Multi-profile gateway multiplexer setup
- runbooks/diagnose-docker — Docker diagnosis
- runbooks/recover-docker-service — Docker service recovery
- runbooks/restart-hermes — Hermes restart procedure
- runbooks/restart-browser — Browser restart procedure
- runbooks/update-hermes-safely — Safe Hermes update
- runbooks/restore-openrouter — OpenRouter restoration
- runbooks/provider-health — Provider health check
- runbooks/troubleshoot-tool-loop — Tool loop troubleshooting
- runbooks/gateway-resource-resilience — Gateway resilience
- runbooks/job-radar — Job Radar search, Notion, dashboard and recovery procedure
Raw
- raw/configs/hermes-config-sanitized.txt — Sanitized config