Files
agent-estate-wiki/current-state.md
T

17 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Current State concept active 2026-07-22 2026-10-01 2026-10-01 2026-10-30 high
system
live-config
live-cron-list
live-hermes-version

Current State

Updated: 2026-10-01

Recent Changes (2026-10-01)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set 4 → 3: cbfe434a3b5e returned to ok as predicted on 2026-09-30. The residual 1a8e90ea2271 dispatch error also cleared — its current failure is a different, upstream problem (below).
  • Telegram delivery failures root-caused and fixed: the target, not the topic, was wrong — a bare deliver: "telegram" resolves to the home DM (1793951355), where group topic IDs cannot exist, so sends hard-failed and the message was lost. 1b23c042e17d and c8d52904e1d6 retargeted to telegram:-1004321904721:102 and verified. Three jobs still carry bare telegram targets (0e753e9720fd, fad5a86979d4, 066bb5191280). See systems/messaging-integrations.
  • Three jobs pinned to models that no longer exist upstream — cannot self-recover: 1a8e90ea2271 → qwen3.5:397b retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); dca8482e4f76 → gpt-5.6-terra plan-gated (403 MODEL_NOT_IN_PLAN, streak 5); fad5a86979d4 → nvidia/…/deepseek-v4-flash absent from the live catalog (HTTP 400, streak 3). Repin to the smart-route alias. See systems/model-providers.
  • GitHub Trending Radar — revoked Telegram bot token: c0a9fa314542 builds its digest then fails the push (getMe → 401). Token in /home/hermes/workspace/github-trending-radar/config.json needs replacing from @BotFather.
  • Memory hygiene pass (2026-09-30): 36 gateway background-process notifications had been written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts hermes_cron_model_default and the job-radar model corrected. See systems/mnemosyne.
  • Provider stack rebuilt from live config: 5 custom providers (omni-101, Omniroute:20128, litellm, command-code, guacano). The provider key custom:omni no longer exists; two cron jobs (e530c89f2be0, d40085da631c) still name it but have not failed, because the default custom block resolves to the same base URL.
  • Vision routing — request recorded, not applied: Anthony asked on 2026-09-30 that vision calls use the local Ollama at http://anthonys-mac-mini:11434 (gemma4). Live config still points at nous / stepfun/step-3.7-flash:free as at 2026-10-01.

Recent Changes (2026-09-30)

  • Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST: Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause: the uv-managed runtime venvs (10 under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv) were missing dotenv and mnemosyne, so the worker died on import before acknowledging the job (ModuleNotFoundError: No module named 'dotenv' — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (checks/repairs all managed venvs via pip install --target), run every 15 min by job 96eb95ce12f2. weekly-save-restricted-bot-restart script restored 19:17; that job is ok again. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • Hermes upgraded since last audit: v0.19.0 (2026.7.20) → v0.21.5+3837.gbfda74c (2026.9.24), upstream 27062c34, update available. Skills 330 → 341.
  • Provider routing drift: default provider is custom → http://100.78.63.20:20128 (OmniRoute) with default model smart-route; auxiliary curator openai-codex / gpt-5.6-terra. Verified live 2026-09-30.
  • Open — Daily Marketing Job Radar blocked: job dca8482e4f76 pinned to custom:omni-101 / gpt-5.6-terra; upstream command-code returns 403 MODEL_NOT_IN_PLAN. 5 consecutive failures (2026-09-26 → 30). See runbooks/job-radar.
  • Open — Telegram thread delivery failures (root cause found 2026-10-01): telegram:1793951355 thread 34795 "not found" for jobs fad5a86979d4 and 1b23c042e17d; c8d52904e1d6 reports "Message thread not found". Cause was the delivery target, not the topic — a bare telegram resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
  • Wiki tree structure confirmed (drift unresolved): /home/hermes/workspace/agent-estate-wiki is the Gitea/Quartz source (this repo); /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault (pages under wiki/, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
  • Multi-bot group chat question raised: Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.

Recent Changes (2026-09-02)

  • SFTPGo outage resolved (02:18 AWST): Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's tailscaled had died, breaking tailnet reachability; PVE's pct start then failed because 5 stale lxc-attach probes from earlier diagnostics pinned CT299's cgroup (/sys/fs/cgroup/lxc/299/ns/.lxc). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), pct stop 299 --skiplock + pct start 299, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: tailscale ping sftpgo → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0. See incidents/2026-09-02-sftpgo-outage.

Attested facts

  • Config version: _config_version: 46 — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously recorded as 33 on 2026-07-25)
  • Default profile: default — Verified, WebUI prompt, 2026-07-22
  • Timezone: Australia/Perth — User-provided + Verified in config, 2026-07-22
  • Main provider: custom → http://100.78.63.20:20128 (OmniRoute) — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously custom:litellm → litellm:4000, then custom:omni → omni.kangaroo-eel.ts.net)
  • Default model: smart-route — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously freellmapi-auto)
  • Fallback providers: opencode-go / mimo-v2.5, opencode-zen / deepseek-v4-flash-free, nous / stepfun/step-3.7-flash:free, ollama-cloud / qwen3.5:397b, kilocode / poolside/laguna-m.1:free, custom / litellm:4000 — Verified, config, 2026-09-30
  • Auxiliary curator model: openai-codex / gpt-5.6-terra — Verified, config, 2026-09-30
  • Memory backend: Mnemosyne — Verified, config, 2026-07-22
  • Memory enabled: true — Verified, config, 2026-07-22
  • Streaming enabled: true — Verified, config, 2026-07-22
  • Dashboard public URL: https://hermes.martinwa.org/ — Verified, config, 2026-07-22
  • Pet config: enabled/slug null — Verified, config, 2026-07-22
  • WebUI connected platforms: local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22

Recent Changes (2026-08-02)

  • Browser backend reverted: Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. browser.cloud_provider: camofox, camera_provider: camofox. Steel plugin disabled. See comparisons/steel-vs-camoufox and decisions/rollback-steel-to-camofox.
  • Mnemosyne capture fix (2026-08-01): Automatic conversational capture was broken by stringified YAML (sync_roles: "['user']"). Fixed by rewriting structured Mnemosyne settings with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
  • Mnemosyne vector coverage gap discovered: Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
  • DeepSeek V4 Flash 0731: Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (^[a-zA-Z0-9_-]+$) — may cause issues with OmniRoute-transformed tool names.
  • Uber Eats report upgraded: Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with camofox-browser-automation + browser-backend-verification skills.
  • GitHub Trending Radar — Daily Digest removed: Stopped by user on 2026-08-02. Weekly roundup still active.
  • Ops profile renamed: Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
  • Wiki audit cron job created: Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
  • Ops profile Mnemosyne enabled: Mnemosyne plugin enabled on ops profile with profile_isolation: true — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: ~/.hermes/profiles/ops/config.yaml (memory.mnemosyne.profile_isolation: true, allow_tool_override: false).
  • OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04): Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.
  • OmniRoute three-layer fix (2026-08-07): npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See systems/omniroute.
  • Nanobot status degraded (2026-08-07): CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See systems/nanobot.
  • CT460 memory bumped (2026-08-07): 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.

Recent Changes (2026-08-08)

  • Cron job model repinning: 3 jobs repinned from dead freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker repinned from dead freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named cron jobs (e.g. vertex/gemini-2.5-flash, gpt-5.6-terra, deepseek-v4-flash-free) repinned to smart-route. Daily Marketing Job Radar swapped to gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build now serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, and CT333 Nanobot has a live jobs.json with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.

Recent Changes (2026-08-09)

  • Portainer password reset (CT245): Admin user is anthony (not admin). Password was forgotten; reset via docker run --rm -v portainer_data:/data portainer/helper-reset-password after stopping the Portainer container. Verified end-to-end with POST /api/auth → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on :9443 (see systems/homepage-dashboard). Recovery procedure: stop container → run helper image → restart → verify with auth API.
  • SFTPGo security review (CT299): Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path // resolves to / — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is root, (4) quota value F1 on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.

Recent Changes (2026-08-10)

  • Cron job failures — session storage issue: Four cron jobs failed on 2026-08-10 with two distinct error patterns:
    • TimeoutError (idle for 601–602s): "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
    • RuntimeError (session storage): "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
    • Root cause: Disk at 83% (11GB free on 65GB). state.db is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
    • Impact: News digests and GitHub Trending did not deliver. Job Radar did not run.
    • See: systems/scheduled-tasks Known Issues section.
  • OpenCode WebSocket proxy errors: Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, opencode serve), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.

Recent Changes (2026-08-14)

  • OmniRoute factory reset on CT601: Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at /opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See systems/omniroute.
  • Hermes default model changed: auto/best-free → resilient-free in ~/.hermes/config.yaml. auto/best-free was misconfigured (no combo definition, routed to paid gpt-5.6-sol).
  • Multiple free combos broken by upstream model ID changes: OpenRouter Poolside laguna-s-2-1:free → laguna-s-2.1:free (dash→dot); Groq llama-4-scout-17b-16e-instruct retired. Affected: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free.
  • Guanaco router updated: CT205 ollama container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See systems/guanaco.
  • GPT-5.6 Luna research: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See systems/model-providers.
  • Hermes UI restart: Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
  • LiteLLM routing issue identified: Hermes was previously pointed at LiteLLM (http://litellm:4000/v1, model free-auto) causing 6–26s per call. OmniRoute auto/best-free tested at 265ms. Config now points at OmniRoute.

Recent Changes (2026-08-13)

  • OmniRoute migrated CT600 → CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Backup: ~/.hermes/config.yaml.bak.omniroute-removal. See systems/omniroute.
  • Evening stand-down briefing weather bugfix: Weather collector returns a plain string (+16°C Sunny), but the three dashboard widgets (evening.html, evening-grid.html, evening-briefing.html) read object fields → undefined. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.