110 lines
9.7 KiB
Markdown
110 lines
9.7 KiB
Markdown
# Anthony's Systems Wiki
|
|
> Operational knowledge for Anthony's agents, infrastructure,
|
|
> model routing and automation. Last updated: 2026-10-18
|
|
>
|
|
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
|
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
|
> Guest IDs collide between the two hosts.
|
|
|
|
## Start here
|
|
- [[current-state]] — Current systems status
|
|
- [[systems/hermes-agent]] — Hermes Agent overview and version
|
|
- [[systems/current-profile]] — Active Hermes profile details
|
|
|
|
## Systems
|
|
### Core
|
|
- [[systems/hermes-agent]] — Agent version, install, config paths
|
|
- [[systems/current-profile]] — Active profile details
|
|
- [[systems/nextcloud]] — Nextcloud file sharing (CT270)
|
|
- [[systems/guanaco]] — Guanaco LLM proxy on CT205 (Ollama stack)
|
|
- [[systems/model-providers]] — LiteLLM + OmniRoute providers, routing setup
|
|
- [[systems/memory-backend]] — Mnemosyne vector memory, settings
|
|
- [[systems/mnemosyne]] — Mnemosyne cron jobs and config evidence
|
|
- [[systems/omniroute]] — OmniRoute catalog, combos, and host path
|
|
- [[systems/openrouter]] — OpenRouter current state and history
|
|
- [[systems/opencode-go]] — OpenCode Go subscription, model limits, pricing
|
|
- [[systems/nanobot]] — Nanobot current state and history
|
|
- [[systems/browser-backend]] — Camofox (primary, `systemd --user` on CT460, REST `:9377`) + Steel (disabled); memory-containment drop-in and the 2026-10-12 ceiling event documented
|
|
- [[systems/hermes-ui]] — Browser Hermes Desktop (przbadu/hermes-ui)
|
|
- [[systems/paseo]] — Paseo coding-agent daemon + web terminal workspace (CT460, :6767)
|
|
- [[systems/ignis]] — Ignis browser-native Obsidian (wiki interface)
|
|
- [[systems/terminal-backend]] — Shell persistence, working directory
|
|
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
|
|
|
### Integrations
|
|
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
|
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **8 carry a non-clean `last_status`** as of 2026-10-18 00:3x — down from 12. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
|
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ **2026-10-17** stocktake slot failed (`HTTP 503`), so `freshness-20261017.txt` is missing and the index is stamped **10-16**; the **10-18 03:00** slot closes it. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
|
|
|
## Incidents
|
|
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.**
|
|
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
|
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
|
- [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
|
|
- [[incidents/2026-10-05-dead-model-pin-repin]] — Seven cron jobs on dead model pins (`cliprox/terra`, `qwen3.5:397b`, `deepseek-v4-flash`, `gpt-5.6-terra`) repinned to routing aliases on `custom:omni-101`; failing set 7 → 0 (resolved 2026-10-05)
|
|
- [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
|
|
- [[incidents/2026-10-03-ct450-hermes-update]] — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
|
|
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); **closed 2026-10-05** when all three were repinned off the pin
|
|
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
|
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. **Recurred 2026-10-10 inside the *gateway* runtime env** — `mnemosyne` and `elevenlabs` missing from the committed uv env the gateway loads; both repaired and verified the same evening
|
|
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (Aug 2026, **superseded**): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
|
|
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
|
|
|
### Monitoring & Dashboards
|
|
- [[systems/homepage-dashboard]] — Homepage dashboard (CT201)
|
|
- [[systems/home-assistant]] — Home Assistant integration
|
|
|
|
### Reports
|
|
- [[reports/uber-eats/index]] — Uber Eats monthly spend reports
|
|
- [[reports/communities-wa-social-media-audit-2026]] — Department of Communities WA social media audit and first-day brief
|
|
- [[systems/obsidian]] — Obsidian vault structure and conventions
|
|
- [[queries/retained-research-answers]] — Retained research answers (fastest VNC clients, 2026-10-11)
|
|
|
|
## Infrastructure
|
|
### Services
|
|
- [[infrastructure/hosts]] — Hosts, containers, important paths
|
|
- [[infrastructure/docker-services]] — Docker compose files
|
|
- [[infrastructure/credentials-map]] — Where credentials live (no values)
|
|
- [[infrastructure/network-map]] — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel
|
|
- [[infrastructure/domains-and-tunnels]] — martinwa.org hostnames and Cloudflare Tunnel routing
|
|
- [[infrastructure/backups]] — hermes backup CLI, retention, and gaps
|
|
|
|
### Networking
|
|
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14)
|
|
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **45-node** tailnet inventory (2026-10-14)
|
|
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
|
|
|
## Concepts
|
|
- [[concepts/agent-memory]] — Memory model
|
|
- [[concepts/model-routing]] — Routing concepts
|
|
- [[concepts/persistent-shells]] — Terminal persistence
|
|
- [[concepts/browser-automation]] — Browser automation concepts
|
|
- [[concepts/tool-calling]] — Tool calling patterns
|
|
|
|
## Decisions
|
|
- [[decisions/index]] — Decision index
|
|
- [[decisions/keep-openrouter-as-fallback]] — Keep OpenRouter as fallback provider
|
|
- [[decisions/persistent-shell-enabled]] — Enable shell persistence
|
|
|
|
## Comparisons
|
|
- [[comparisons/camoufox-vs-chrome]] — Camoufox vs Chrome
|
|
- [[comparisons/hermes-vs-nanobot]] — Hermes vs Nanobot
|
|
- [[comparisons/memory-backends]] — Memory backend comparison
|
|
- [[comparisons/model-routers]] — Model router comparison
|
|
|
|
## Runbooks
|
|
- [[runbooks/multiplexer-setup]] — Multi-profile gateway multiplexer setup
|
|
- [[runbooks/diagnose-docker]] — Docker diagnosis
|
|
- [[runbooks/recover-docker-service]] — Docker service recovery
|
|
- [[runbooks/restart-hermes]] — Hermes restart procedure
|
|
- [[runbooks/restart-browser]] — Browser restart procedure
|
|
- [[runbooks/update-hermes-safely]] — Safe Hermes update
|
|
- [[runbooks/restore-openrouter]] — OpenRouter restoration
|
|
- [[runbooks/provider-health]] — Provider health check
|
|
- [[runbooks/troubleshoot-tool-loop]] — Tool loop troubleshooting
|
|
- [[runbooks/gateway-resource-resilience]] — Gateway resilience
|
|
- [[runbooks/job-radar]] — Job Radar search, Notion, dashboard and recovery procedure
|
|
|
|
## Raw
|
|
- [[raw/configs/hermes-config-sanitized.txt]] — Sanitized config
|