Files
agent-estate-wiki/index.md
T
hermes-audit ebf57146f1 audit(2026-10-13): 10-12 clean (174/0, 3rd day) but GitHub Trending Radar token DEAD (getMe 401) - supersedes the wrong 2026-10-06 'refuted' entry
Incidents 5->2 (deepseek-v4-flash pair closed 03:05:39, Job Radar timeout 07:33:41); storm page re-tallied 11/12 closed with closure timeline.

NEW: Camofox hit its cgroup ceiling inside the 07:01 Job Radar run (2.5G, swap exhausted, 383 tasks) - containment drop-in documented; browser-backend/steel-vs-camoufox/browser-automation corrected to REST :9377 on CT460 (was :9093 on CT450).

Mnemosyne 96.7%/99.4% coverage green; 18 cron sessions, 0 interactive; retention floor 2026-10-07.
2026-10-13 00:19:15 +08:00

8.1 KiB

Anthony's Systems Wiki

Operational knowledge for Anthony's agents, infrastructure, model routing and automation. Last updated: 2026-10-13

Start here

Systems

Core

Integrations

  • systems/messaging-integrations — Telegram, Discord, email, Slack, etc.
  • systems/scheduled-tasks — 29 cron jobs (26 enabled, 3 disabled); 3 carry a non-clean last_status as of 2026-10-13 00:15. Three consecutive clean days: 2026-10-12 ran 174 completed / 0 failed (2026-10-11 finalised 172 / 0), and open incidents fell 5 → 2 on 2026-10-12 (three closed: both deepseek-v4-flash rows + the Job Radar timeout). The DHL monitor 62619aa6bcb3 was retired on 2026-10-10. ⚠ The 2026-10-12 clean sheet is not an all-clear — GitHub Trending Radar's Telegram token is revoked (getMe → 401) while the cron row stays green. See incidents/2026-10-09-cron-provider-timeout-storm
  • systems/skills-index — Installed skills inventory (346 skills; auto-regenerated daily by the 03:00 stocktake job — verified 2026-10-13 against the stocktake's own 0eee5f1 commit: 346 skills / 70 categories). Note: 347 SKILL.md files exist on disk — the generator de-dups one nested duplicate, see systems/scheduled-tasks; regenerated daily — the 10-10, 10-11 and 10-12 stocktake runs all committed)

Incidents

  • incidents/2026-10-09-cron-provider-timeout-storm — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST; 11 of its 12 incidents are now closed (five on 2026-10-10, the last hold-out dca848_23fcd2b18aaa on 2026-10-12 07:33:41). 1 remains alerted — 1b23c042e17d, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
  • incidents/2026-10-09-memory-recall-relevance — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved)
  • incidents/2026-10-05-dead-model-pin-repin — Seven cron jobs on dead model pins (cliprox/terra, qwen3.5:397b, deepseek-v4-flash, gpt-5.6-terra) repinned to routing aliases on custom:omni-101; failing set 7 → 0 (resolved 2026-10-05)
  • incidents/2026-10-03-mnemosyne-vector-recall-fts-only — Mnemosyne recall had been scoring keyword-only (fastembed missing from the managed runtimes); repaired 2026-10-03, coverage re-verified at 96.5% working / 99.4% episodic
  • incidents/2026-10-03-ct450-hermes-update — Chaos instance CT450 updated v0.19.0 → v0.21.5 (resolved); access path and pre-flight checks documented
  • incidents/2026-10-01-cliproxy-connection-auth-expired — OmniRoute cliprox/terra connection credential expired (all 3 jobs on the pin failed, settled 2026-10-04); closed 2026-10-05 when all three were repinned off the pin
  • incidents/2026-08-26-work-outlook-basic-auth-disabled — Work Outlook mailbox unreachable: tenant allows only XOAuth2, himalaya configured for password (open since 2026-08-26)
  • incidents/2026-09-28-hermes-managed-runtime-deps — Managed-runtime venvs missing dotenv/mnemosyne broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min. Recurred 2026-10-10 inside the gateway runtime env — mnemosyne and elevenlabs missing from the committed uv env the gateway loads; both repaired and verified the same evening
  • incidents/2026-08-18-mnemosyne-capture-pipeline — Mnemosyne capture pipeline broken (Aug 2026, superseded): capture verified healthy 2026-10-12 — 786 working rows, +2 that day, 98.3 % embedded
  • incidents/2026-08-15-skills-index-drift — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration

Monitoring & Dashboards

Reports

Infrastructure

Services

Networking

Concepts

Decisions

Comparisons

Runbooks

Raw