Files
agent-estate-wiki/index.md
T

24 KiB
Raw Blame History

Anthony's Systems Wiki

Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-10-26

⚠ Two Proxmox hosts exist — always qualify a guest with its host. A second unclustered hypervisor, m93p (100.104.51.98, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. Guest IDs collide between the two hosts.

Start here

Systems

Core

Integrations

  • systems/conduit — Conduit: Hermes on Anthony's phone (dashboard plugin + iOS app — voice calls, cards, push). Plugin tree ~/.hermes/plugins/conduit_push, v0.69.2 (HEAD 942ddb8, 2026-10-24 15:35 — "Pulse: list candidates, decide in code, journal them (#342)"; 24 commits on since the 0.64.0 baseline, 0 behind origin/main), upstream kaishi00/hermes-conduit-notifier. Gateway on :9119 restarted 2026-10-25 18:42:33 — so the upstream commits are now LIVE (the 10-24 "not live" note is superseded). One registered device (iPhone app 2.0.0). Still carries the uncommitted in-tree dashboard/plugin_api.py patch; conduit-calls.json → history is empty, so the memory fix still has no positive call evidence, but conduit_show ran once on 10-25 14:37 (card delivered). ⚠ Known failure: three 10-24 requests to be phoned were refused by the agent, which never called the tool; no recurrence since → incidents/2026-10-24-conduit-call-request-refused.
  • systems/messaging-integrations — Telegram, Discord, email, Slack, etc. Mattermost's long-running red now has a cause: CT255 is stopped — still stopped on 2026-10-26 (peer offline 36 d, tx 312 / rx 0; host answers neither :8065 nor ICMP). 268 connect timeouts on 10-25 vs 264 on 10-24; the reconnect attempt counter reset to 54 at the 19:27:13 gateway restart — a per-process number, not an outage length. No cron job delivers via Mattermost.
  • systems/push-notifications — Cadu/Vory push bridge (hermes-push, vory-push): gateway on :9119, state in ~/.hermes/push/. Broken — connected: false (the bridge dials a name that resolves to 127.0.1.1 where nothing serves 443); chronic, not a blip.
  • systems/scheduled-tasks — As of 2026-10-26 00:1x: 28 jobs (25 enabled, 3 disabled), 5 non-clean last_status, unclosed incident rows still 2; 2026-10-25 settled 172 / 2 and 2026-10-24 settled 168 / 0. ⚠ Both 10-25 failures are 08eb86f02d37 (Gmail Action Digest), which also fired six times that day — and neither raised an incident row → incidents/2026-10-25-gmail-digest-runaway-and-refire. ⚠ The global default reverted to custom:omni-101 / smart-route on 2026-10-24 19:06, so the 8 unpinned jobs are back on OmniRoute and the fleet is a single path again → systems/current-profile. ⚠ Nous Portal credentials still invalid — all 8 auxiliary slots moved to no-think/freeapi/auto at 19:13 → incidents/2026-10-22-nous-portal-refresh-token-invalid. Three gateway starts on 10-25 (00:32:20, 18:30:04, 19:27:13). Prior window — as of 2026-10-25 00:1x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, unclosed incident rows still 2, and 2026-10-24 closed 167 / 0 — the third consecutive clean day (10-22 → 10-24); 2026-10-23 settled at 170 / 0. Zero incident rows opened or closed in-window; pins unchanged. ⚠ Off-cron fault, open: three Conduit call requests were refused by the agent on 2026-10-24 12:53–12:56, with zero tool calls → incidents/2026-10-24-conduit-call-request-refused. Prior window: As of 2026-10-24 00:2x: 28 jobs (25 enabled, 3 disabled), 4 non-clean last_status, and 2026-10-23 closed 169 / 0 — the third zero-failure day in four. The 2026-10-09 provider timeout storm is now fully closed: 1b23c042e17d ran clean on its 10-23 08:04 slot, closing the storm's last two rows, so all 12 are closed and unclosed incident rows fell 4 → 2. ⚠ New off-cron fault, fixed in-window: Conduit voice calls were being abandoned by the memory layer — the plugin hard-coded a 10 s budget for a request that includes Mnemosyne's 10–45 s cold start; three calls ran without recall. Patched to 60 s (CONDUIT_MEMORY_TIMEOUT_S) + degraded-not-aborted on TokenError; dashboard restarted 18:54:43, no abort since. Patch uncommitted → incidents/2026-10-22-conduit-memory-start-timeout, systems/conduit. The 10-22 stocktake overrun did not repeat (10-23 03:07 committed 250c735). Prior window: As of 2026-10-22 00:1x: 28 jobs (25 enabled, 3 disabled), 9 non-clean last_status, and 2026-10-21 closed 171 / 1 — where the single failure is a failure mode this wiki had not seen. dca8482e4f76 (Daily Marketing Job Radar) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip. 08eb86f02d37 cleared (08:30 slot clean; incident resolved 08:37:21) and 2026-10-20 settled at 169 / 0 + 1 unknown. ⚠ Hermes Agent was updated to 0.22.3 at 22:47 → 22:57 with its pre-update snapshot skipped → systems/hermes-agent. Prior window: 2026-10-20 closed 168 / 0 — the first zero-failure day since 10-12. The omni-101 503 fault did not recur (back to monitoring) and 4 incident rows closed → 9 unclosed. ⚠ Two new findings: the Gitea config backup has not pushed since 2026-08-08 (75 unpushed commits) while its row reads ok → incidents/2026-08-08-gitea-config-backup-push-dead; and a gateway unclean exit on 2026-10-20 orphaned a digest run — no delivery row, no incident → incidents/2026-10-20-gateway-unclean-exit-orphaned-run. Also: the global model default moved to nous direct (meituan/longcat-2.5-preview:free) on 2026-10-20, taking the 8 unpinned jobs off OmniRoute while 20 of 28 still pin it → systems/current-profile. Prior window: 28 cron jobs (25 enabled, 3 disabled); 12 carry a non-clean last_status as of 2026-10-20 00:09 — up from 7, because the omniroute-1 503 resource-pressure fault returned on 2026-10-19 (second burst in four days: 9 model calls across 8 jobs, 06:01 and 07:01 → 12:01, all model: smart-route on custom:omni-101, while /api/health stayed 200 ok) → incidents/2026-10-16-omniroute-503-resource-pressure. 2026-10-19 closed 159 / 13 against 10-18's 168 / 1. The Mosquito Tracker was removed (29 → 28 jobs), and the 2026-10-18 degenerate-stream placeholder-delivery mode remains an open question → incidents/2026-10-18-degenerate-stream-canned-substitute. Earlier: the connection-refused storm (2026-10-14 → 10-15) — 24 model-call failures across 12 jobs, ~34 h of digests blanked. All 18 model-calling jobs still sit on custom:omni-101 with no failover.
  • systems/skills-index — Installed skills inventory (306 skills / 70 categories as of the 2026-10-23 03:07 stocktake; regenerated daily by the 03:00 job). The generator now excludes _archive/ — the earlier 346/347 counts included it. Freshness at that run: 2 stale / 94 fresh / 6 no-date of 102 pages (freshness-20261023.txt). Note: SKILL.md files on disk still exceed the index (the generator de-dups one nested duplicate; see systems/scheduled-tasks). ⚠ The 2026-10-22 slot ran 97 m 58 s (vs 16 m 01 s on 10-21), finished completed with delivery suppressed and committed nothing; the 10-23 slot closed that hole.

Incidents

  • incidents/2026-10-25-gmail-digest-runaway-and-refire — The Gmail Action Digest ran 1 h 55 m and fired six times on 2026-10-25. The 08:30 slot started 08:31:14 and closed 10:26:37 failed on RuntimeError: Operation interrupted: waiting for model response (347.2s elapsed) — against 91 s (10-23) and 104 s (10-24). The same job then fired five more times (17:19:36, 18:40:06, 19:18:41, 19:30:31, 19:38:53); the 19:18:41 fire died on a new signature — Cron admission missing: the owner never admitted this fire (distinct from the 10-20 orphan) — and all six fires, failures included, carry delivery_outcome: delivered, so up to six digests reached Anthony. Neither failure raised a cron_incidents row; the job now reads last_status: delivery_queued with an empty last_error. Hypothesis (unconfirmed): the 08:31 fire adopted an unrelated long-lived session — its worker made 70 terminal calls on VM101 /dev/sdb backup work with context growing 66 k → 148 k tokens, while context compression was unavailable all day (invalid_grant). → systems/scheduled-tasks, incidents/2026-10-22-nous-portal-refresh-token-invalid

  • incidents/2026-10-22-nous-portal-refresh-token-invalid — Invalid refresh token — Nous Portal runtime credentials are dead, so context summarisation failed. Gateway-log counts: 1 line 2026-10-22, 3 on 10-24, 20 on 10-25 (last 14:53:04, agent.context_compressor: Failed to generate context summary), 0 since the 19:27:13 restart. ⚠ Masked, not fixed: at 2026-10-25 19:13 all eight auxiliary: slots (approval, compression, curator, mcp, profile_describer, skills_hub, title_generation, vision) moved off nous to no-think/freeapi/auto on custom:omni-101 (fallback no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free, free_only: true; compression timeout 90 s, vision 180 s). No hermes model re-authentication has been run and nous is still the first entry in fallback_providers, so failover tries a dead credential. → systems/model-providers, systems/current-profile

  • incidents/2026-10-24-conduit-call-request-refused — The agent refused to phone Anthony, three times, on 2026-10-24 (12:53–12:56). He asked to be called to test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME); all three replies asserted the agent has no calling capability, with zero tool calls in the session. conduit_call_user (toolset conduit) exists, is registered for every session, is reachable via tool_search, and the plugin ships the calling-the-user skill; his call settings (enabled / when_asked / decides) were permissive and the device was paired. Root cause: capability asserted instead of discovered — the first reply came from kilo-auto/free, but the switch to smart-route repeated it. Same audit also corrects the 10-22 incident's closing evidence: conduit-calls.json → ends records turn-ends that fire a watch, not calls, so “two calls closed cleanly 00:03:23 / 00:06:57” were the two Mnemosyne cron sessions — history (real calls) is empty and no call has been placed since the memory fix. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout

  • incidents/2026-10-22-conduit-memory-start-timeout — Conduit voice calls were silently running without memory, 2026-10-22 → 10-23. The phone plugin gave one request — including the Mnemosyne provider start — a hard-coded 10 s budget (MEMORY_TIMEOUT_S), while Mnemosyne's cold start for a conduit-voice session runs 10–45 s. Three calls were abandoned at the memory layer (Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call: 10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16) and answered with no recall of prior conversation. Fixed 10-23 in the plugin's working tree: default 60 s (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that logs and proceeds without recall instead of aborting; dashboard restarted 18:54:43 and no abort line exists after it. ⚠ Corrected 2026-10-25: the “two calls closed cleanly 00:03:23 / 00:06:57 on 10-24” was a false positive — those were the two Mnemosyne cron turn-ends of that night, and conduit-calls.json → history (real calls) is empty, so the fix has no positive call evidence and no call has been placed since. The patch is uncommitted — a plugin update reverts it. The cold start itself was not optimised. → systems/conduit, systems/mnemosyne

  • incidents/2026-10-21-omniroute-pre-dispatch-filter-skip — A third failure signature from the custom:omni-101 / smart-route pin, 2026-10-21. The Daily Marketing Job Radar (dca8482e4f76, 07:00 weekday) died at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — no 503, no connection error, no repetition loop, and zero 503 rows exist for that day (last 2026-10-19 09:02). Its 07:00 sibling 85edf6772bca completed normally, so the node was serving; the run wrote no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). Incident dca848_014affeb6954 was closed 2026-10-22 07:59:22 when the next daily slot ran clean. Not caused by the 22:47 hermes update — that landed 15 h later. → systems/omniroute

  • incidents/2026-10-20-gateway-unclean-exit-orphaned-run — The gateway exited uncleanly on 2026-10-20 (09:09:01, suspected_oom=False) and orphaned a live cron run. 08eb86f02d37 (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed unknown — “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” No delivery row exists for the slot and no incident row was created; the job table keeps showing the previous slot's 503. 20 unclean exits since 2026-08-01 (5 clustered on 2026-08-14/15); this one ended a 27-day clean run. Open: orphan-recovery on restart, and treating a delivery-targeted unknown as a probable non-delivery.

  • incidents/2026-08-08-gitea-config-backup-push-dead — Green row, dead job, 2.5 months: the daily Hermes Config Backup to Gitea has not pushed since 2026-08-07 11:00:47. e27da6c34bbb reads last_status: ok and its executions read completed every night while 75 commits sit unpushed (fatal: could not read Username for 'https://gitea.kangaroo-eel.ts.net' — HTTPS remote, no credential, no helper). Telegram failure notices have fired daily (message_id 8447, 2026-10-20 11:00); AgentMail failed. No off-box ~/.hermes backup in 2.5 months. Second instance of the pattern after the GitHub Trending Radar token revocation. → infrastructure/backups

  • incidents/2026-10-18-degenerate-stream-canned-substitute — The repetition-loop failure mode, 2026-10-18: the FreshRSS Midday run (ecc1fc13655a) wrote "Your request was not processed…", delivered it to topic :346 at 12:08:11, then died at 12:08:25 on RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop. A delivered message is not evidence of delivered content — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? Second surface (2026-10-20 18:37): an interactive desktop session received the same canned line after Rebuilt-message restart limit (3) exceeded → reason=rebuilt_restart_limit_exceeded … response_len=0, so the substitute is emitted by the harness's failure path generally, not only by the repetition-loop guard. The user's question in that session — Nous OAuth dashboard auth plus a gateway token — was never answered.

  • incidents/2026-10-16-omniroute-503-resource-pressure — omniroute-1 load-sheds model calls with HTTP 503 … resource pressure — two bursts: 2026-10-16 → 10-17, and again 2026-10-19. Burst 1: 9 model calls across 7 jobs over ~19 h, self-cleared (reported then as 10-17 = 162 / 5; the DB reads 163 / 5 — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: 9 model calls across 8 jobs on 2026-10-19 (06:01 + 07:01 → 12:01, whole day 159 / 13). /api/health read 200 ok throughout both. Status: active → back to monitoring on 2026-10-21 — 2026-10-20 ran 168 / 0 with zero rows carrying the resource-pressure string, so the second burst is over; it stays a standing risk until the single-node topology changes. 4 rows closed on 10-20 (dca848_867d9caa96a8, 85edf6_256c5e150bc4, ecc1fc_c9976ea24553, ecc1fc_21a72aa11cc4); unclosed rows 13 → 9 (8 alerted / 1 detected); the ledger still tags every 503 row failure_type: unknown, so triage on error text. Single-node topology still open — and every job that failed reads model: smart-route, provider: custom:omni-101.

  • incidents/2026-10-14-omniroute-node-connection-refused-storm — Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15: omniroute-1 (100.78.63.20:20128, provider custom:omni-101) refused TCP connections, so every model-calling cron job failed on every slot — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. All 18 enabled model-calling jobs ride this one node with no failover. Recovered 10-15 22:01 (resolved)

  • incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST. CLOSED — all 12 incident rows are now resolved; the last two (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b, both on the weekly Friday job 1b23c042e17d) closed 2026-10-23 08:04 when that slot ran clean, and the job reads ok again

  • incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)

  • incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (cliprox/terra, qwen3.5:397b, deepseek-v4-flash, gpt-5.6-terra) repinned to routing aliases on custom:omni-101; failing set 7 → 0 (resolved 2026-10-05)

  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic

  • incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented

  • incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute cliprox/terra connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin

  • incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only XOAuth2, himalaya configured for password (open since 2026-08-26)

  • incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing dotenv/mnemosyne broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env — mnemosyne and elevenlabs missing from the committed uv env the gateway loads; both repaired and verified the same evening

  • incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded

  • incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration

Monitoring & Dashboards

Reports

Infrastructure

Services

Networking

  • infrastructure/proxmox — PVE hosts (two: pve 8.x + m93p 9.2.0, unclustered, IDs collide), VMs, LXC containers, storage (/mnt/ssd0 88.17 % / ~63 GiB free, 2026-10-14)
  • infrastructure/tailscale — Tailscale serve routes (CT460 verified exactly) and the 48-node tailnet inventory (re-verified 2026-10-19: guest-vps, openmausbot, openmausbot-vm added; mattermost offline 29d)
  • systems/cloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly

Concepts

Decisions

Comparisons

Runbooks

Raw