Files
agent-estate-wiki/index.md
T
Hermes Rhino 95fd7d33f1 audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day
- burst-2 table names/schedules/aliases corrected from live jobs.json
- non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated)
- 2026-10-17 tail corrected 162/5 -> 163/5
2026-10-20 00:12:39 +08:00

12 KiB

Anthony's Systems Wiki

Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-10-20

⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor, m93p (100.104.51.98, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.

Start here

Systems

Core

Integrations

  • systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped (peer offline 29d; host answers neither :8065 nor ICMP).
  • systems/push-notifications — Cadu/Vory push bridge (hermes-push, vory-push): gateway on :9119, state in ~/.hermes/push/. Broken — connected: false (the bridge dials a name that resolves to 127.0.1.1 where nothing serves 443); chronic, not a blip.
  • systems/scheduled-tasks — 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-clean last_status as of 2026-10-20 00:09 — up from 7, because the omniroute-1 503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, all model: smart-route on custom:omni-101, while /api/health stayed 200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit on custom:omni-101 with no failover.
  • systems/skills-index — Installed skills inventory (346 skills / 70 categories; regenerated daily by the 03:00 stocktake job). Note: 347 SKILL.md files exist on disk — the generator de-dups one nested duplicate, see systems/scheduled-tasks. The 2026-10-17 slot failed (HTTP 503), but the 10-18 03:00 slot closed the hole: freshness-20261018.txt written, index regenerated and committed (dd53e5f) — freshness 47 stale / 41 fresh / 7 no-date. ⚠ The 2026-10-13 regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: updated/verified_on 10-12 → 10-13).

Incidents

  • incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic :346 at 12:08:11, then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route?

  • incidents/2026-10-16-omniroute-503-resource-pressure — omniroute-1 load-sheds model calls with HTTP 503 … resource pressure — two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as 10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13). /api/health read 200 ok throughout both. Status back to active — a self-clearing fault that returns twice in four days is a standing risk, not a closed incident. 4 incident rows opened and 2 auto-closed on 10-19 (a third, ea79a2_747b27556fd0, closed 10-20 00:04); 13 unclosed rows overall (12 alerted / 1 detected); the ledger still tags every 503 row failure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed reads model: smart-route, provider: custom:omni-101.

  • incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15: omniroute-1 (100.78.63.20:20128, provider custom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved)

  • incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST; 11 of its 12 incidents are now closed (five on 2026-10-10, the last hold-out dca848_23fcd2b18aaa on 2026-10-12 07:33:41). 1 remains alerted — 1b23c042e17d, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)

  • incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)

  • incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (cliprox/terra, qwen3.5:397b, deepseek-v4-flash, gpt-5.6-terra) repinned to routing aliases on custom:omni-101; failing set 7 → 0 (resolved 2026-10-05)

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic

  • incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented

  • incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute cliprox/terra connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin

  • incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only XOAuth2, himalaya configured for password (open since 2026-08-26)

  • incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing dotenv/mnemosyne broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env — mnemosyne and elevenlabs missing from the committed uv env the gateway loads; both repaired and verified the same evening

  • incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded

  • incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration

Monitoring & Dashboards

Reports

Infrastructure

Services

Networking

  • infrastructure/proxmox — PVE hosts (two: pve 8.x + m93p 9.2.0, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd0 88.17 % / ~63 GiB free, 2026-10-14)
  • infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 48-node tailnet inventory (re-verified 2026-10-19: guest-vps, openmausbot, openmausbot-vm added; mattermost offline 29d)
  • systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly

Concepts

Decisions

Comparisons

Runbooks

Raw