{"schema":{"slug":"schema","filePath":"SCHEMA.md","title":"SCHEMA","links":[],"tags":[],"content":"SCHEMA — Anthony’s Systems Wiki\nPurpose\nHelp Hermes understand and maintain Anthony’s agents, services,\nautomations, model routing and technical decisions.\nRules\n\nDo not store passwords, API keys, recovery codes, or private SSH keys.\nA credentials-map.md records where secrets live; never their values.\nPreserve raw sources unchanged in raw/.\nUpdate existing pages instead of creating duplicates.\nEvery page uses YAML frontmatter and links to at least two related pages.\nSeparate facts from procedures:\nsystem page = what/where/configured;\nrunbook = how to recover;\nconcept = why;\ndecision = what we chose and why;\nraw doc snapshots = official source.\nRecord important choices in decisions/ with revisit triggers.\nCapture repeatable fixes in runbooks/, not articles.\nRecord tests and experiments in experiments/ with explicit verdicts.\nMark outdated material as deprecated; never silently delete.\nInclude dates and version markers on technical instructions.\nAsk before writing more than ten pages at once.\n\nFrontmatter\n---\ntitle: Page Title\ntype: system | infrastructure | concept | decision | comparison | runbook | experiment | query\nstatus: active | experimental | deprecated | unresolved | archived\ncreated: YYYY-MM-DD\nupdated: YYYY-MM-DD\nreview_after: YYYY-MM-DD\nconfidence: high | medium | low\ntags: [from taxonomy]\nsources: [raw/doc]\n---\nTag Taxonomy\n\nsystems: hermes, nanobot, router, memory, browser, automation, infrastructure\ncontent: concept, comparison, decision, runbook, experiment, source\nstatus: active, experimental, deprecated, unresolved, archived\n\nFreshness\nPages older than 90 days without update should be reviewed.\nversion_applies_to: and verified_on: should appear on operational pages."},"agents/hermes-chaos":{"slug":"agents/hermes-chaos","filePath":"agents/hermes-chaos.md","title":"Hermes Chaos","links":["agents/hermes-production","infrastructure/hosts","systems/hermes-agent"],"tags":["hermes","chaos","ct450","experimental","agent"],"content":"Hermes Chaos\nPurpose\nExperimental Hermes instance for testing new configurations, skills, themes, model routing, and plugin changes before promoting them to production. Changes here must not affect CT460 production without explicit approval.\nLocation\n\nContainer: LXC CT450 (hermesagent)\nTailscale IP: 100.98.205.112\nHostname: hermesagent\nSoftware: przbadu/hermes-ui (Vite web app for browser use)\n\nCurrent State\n\nStatus: Active (experimental)\nHermes UI: przbadu/hermes-ui running on CT450\nDuplicate cron jobs: 11 paused (no post-pause execution records since 2026-08-01)\nProduction counterparts: CT460 has 11 enabled jobs with successful latest runs\n\nPhase One Review (2026-08-08)\nPhase One seven-day review completed and independently live-verified:\n\nCT450 duplicate jobs remained paused with no post-pause execution records\nCT460 counterparts remain enabled with successful latest runs\nCT333 schedules remained enabled and healthy\nPhase Two is ready for consideration but is NOT approved\n\nKey Rules\n\nDo not promote changes to production without explicit approval. Chaos is for testing.\nDuplicate cron jobs are paused. Do not re-enable without approval.\nSeparate state. Memory, skills, and config are independent from CT460.\n\nRelationship to Production\nHermes Chaos (CT450) and Hermes Production (CT460) are separate Hermes instances with:\n\nSeparate configs\nSeparate skills\nSeparate cron jobs\nSeparate memory state\nSeparate Tailscale identities\n\nRelated\n\nhermes-production — The production instance\nhosts — Host inventory\nhermes-agent — Hermes Agent system details\n"},"agents/hermes-production":{"slug":"agents/hermes-production","filePath":"agents/hermes-production.md","title":"Hermes Production","links":["systems/model-providers","systems/memory-backend","systems/scheduled-tasks","systems/messaging-integrations","agents/hermes-chaos","agents/nanobot","systems/hermes-agent","systems/current-profile","infrastructure/hosts"],"tags":["hermes","production","ct460","agent"],"content":"Hermes Production\nPurpose\nPrimary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks. This is the production instance — changes here affect live workflows.\nLocation\n\nContainer: LXC CT460 (hermes)\nTailscale IP: 100.96.100.82\nHostname: hermes\nPort: 8082 (HTTP), 9119 (gateway)\nMemory: 14GB (upgraded from 5096MB on 2026-08-07)\n\nCurrent State\n\nStatus: Active\nHermes version: v0.19.0\nPython: 3.11.15\nInstall directory: /home/hermes/.hermes/hermes-agent\nConfig: ~/.hermes/config.yaml\nProfile: default (~/.hermes/profiles/default/)\nSoul: Rhino — trickster-mentor in a green jacket 🎩\n\nProfiles\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProfileTelegram ChatsPurposedefault-1004321904721 (Rhino chat)Main agentops-1003914987043 (Elephant ops), -1003932503629 (OPS Tember rescue)Operations/infra agent (Tusk)\nProviders & Routing\n\nDefault provider: custom:litellm → http://litellm:4000/v1\nFallback provider: custom:omniroute → http://omniroute:20128/v1\nDefault model: freellmapi-auto\nSee model-providers for full breakdown.\n\nMemory\n\nBackend: Mnemosyne (local native)\nVector type: int8\nShared surface: data/shared/mnemosyne.db\nSee memory-backend for details.\n\nScheduled Tasks\n\nCron jobs: 25+ active jobs\nDelivery targets: Telegram (primary), Discord, origin, local\nNotable: Birthday checker, job radar, newsletter digests, mnemosyne health/watchdog/backup\nSee scheduled-tasks for full table.\n\nMessaging\n\nTelegram, Discord, Slack, Email, Mattermost, ntfy\nSee messaging-integrations for details.\n\nKey Differences from Chaos\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nAspectProduction (CT460)Chaos (CT450)PurposeStable daily operationsExperimental/testingCron jobs25+ enabled11 paused duplicatesConfig changesRequires verificationSafe to experimentMemoryProduction MnemosyneSeparate state\nRelated\n\nhermes-chaos — The experimental twin\nnanobot — Scout agent on CT333\nhermes-agent — Hermes Agent system details\ncurrent-profile — Active profile details\nhosts — Host inventory\n"},"agents/index":{"slug":"agents/index","filePath":"agents/index.md","title":"Agent Estate","links":["agents/hermes-production","agents/hermes-chaos","agents/nanobot","agents/qwenpaw","agents/zeroclaw","agents/open-claw","systems/omniroute","systems/model-providers","infrastructure/proxmox","infrastructure/tailscale","infrastructure/domains-and-tunnels","systems/hermes-agent","systems/nanobot","infrastructure/hosts"],"tags":["agents","estate","overview"],"content":"Agent Estate\n\nMap of all AI agents, their locations, and their roles in Anthony’s setup.\n\nOverview\nThe agent estate consists of multiple Hermes instances, a QwenPaw deployment, ZeroClaw, Open-Claw, and Nanobot — each with a defined role and home container.\nAgents\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nAgentContainerRoleStatushermes-productionCT460Production Command CentreActivehermes-chaosCT450Experimental Chaos LabActivenanobotCT333Scout (quota notifications, daily briefing)ActiveqwenpawCT465Multi-agent gatewayActivezeroclawCT333Autonomous daemon (TOML config)Activeopen-clawVM 403Autonomous gateway nodeActive\nShared Infrastructure\nAll agents share:\n\nomniroute — model routing and provider management\nmodel-providers — LiteLLM + OmniRoute provider catalog\nproxmox — PVE host and containers\ntailscale — VPN and serve routes\ndomains-and-tunnels — Cloudflare Tunnel routing\n\nRelationships\n\nHermes production (CT460) and Hermes Chaos (CT450) are separate Hermes instances with separate configs, skills, and cron jobs.\nZeroClaw and Nanobot share the same LXC container (CT333) but are different software with different configs (TOML vs JSON).\nOpen-Claw (VM 403) is a completely separate Docker-based gateway, not related to ZeroClaw despite the similar name.\nQwenPaw (CT465) is a separate agent platform with its own workspace and model routing.\n\nKey Rules\n\nNever assume one agent’s config applies to another. Each agent has its own config format, paths, and provider routing.\nCT333 has two agents. ZeroClaw (root, TOML) and Nanobot (JSON) coexist. Do not restart one without checking the other.\nCT450 Chaos is experimental. Changes here must not affect CT460 production without explicit approval.\nPhase Two (estate consolidation) is not approved. CT450 duplicate cron jobs remain paused; CT460 counterparts remain enabled.\n\nRelated\n\nhermes-agent — Hermes Agent system details\nnanobot — Nanobot system details\nhosts — Host and container inventory\n"},"agents/nanobot":{"slug":"agents/nanobot","filePath":"agents/nanobot.md","title":"Nanobot","links":["agents/zeroclaw","agents/open-claw","systems/nanobot","comparisons/hermes-vs-nanobot"],"tags":["nanobot","ct333","scout","agent"],"content":"Nanobot\nPurpose\nScout agent for Anthony’s estate. Handles quota notifications and daily agent briefings. Runs on CT333 alongside ZeroClaw (different software, same container).\nLocation\n\nContainer: LXC CT333 (clawtest)\nTailscale IP: 100.122.189.81\nPort: 8766 (Nanobot), 8900 (previously, now DOWN)\nShares container with: zeroclaw\n\nCurrent State\n\nStatus: Active (scout role)\nCron schedules: Enabled with recent ok status\nQuota notifications: Enabled — kept active per Anthony’s direction\nDaily briefing: Enabled\n\nPhase One Review (2026-08-08)\nCT333 schedules remained enabled and healthy during the seven-day review.\nKey Rules\n\nNanobot schedules remain enabled unless Anthony explicitly approves changes.\nDo not conflate with ZeroClaw. They share CT333 but are different software.\nDo not conflate with Open-Claw. Open-Claw is VM 403, not CT333.\n\nRelationship to ZeroClaw\nNanobot and ZeroClaw share LXC CT333 (clawtest):\n\nNanobot: JSON config, Telegram gateway bot\nZeroClaw: TOML config, autonomous daemon (root)\n\nThey are different software with different configs, different users, and different purposes. Restarting one does not affect the other.\nRelated\n\nzeroclaw — Autonomous daemon on same container\nopen-claw — Separate gateway on VM 403\nnanobot — Detailed system page\nhermes-vs-nanobot — Comparison\n"},"agents/open-claw":{"slug":"agents/open-claw","filePath":"agents/open-claw.md","title":"Open-Claw","links":["agents/zeroclaw","agents/nanobot","infrastructure/proxmox","infrastructure/tailscale"],"tags":["openclaw","open-claw","vm403","gateway","agent"],"content":"Open-Claw\nPurpose\nAutonomous AI agent gateway running on a separate QEMU VM. Docker-based with Tailscale networking, serving a control UI and gateway for AI agent operations. Completely separate from ZeroClaw despite the similar name.\nLocation\n\nVM: QEMU VM 403 (open-claw)\nHostname: open-claw\nLAN IP: 192.168.178.193 (DHCP)\nTailscale IP: 100.64.172.34\nTailscale Domain: open-claw.kangaroo-eel.ts.net\nControl UI: open-claw.kangaroo-eel.ts.net\nOS User: openclaw (sudo-capable)\n\nCurrent State\n\nStatus: Active\nConfig format: JSON\nAuth: Password-based (not paired tokens)\nSoftware: Docker-based (community-scripts template)\n\nKey Differences from ZeroClaw\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nAspectOpen-Claw (VM 403)ZeroClaw (CT333)TypeQEMU VMLXC containerConfig formatJSONTOMLOS useropenclawrootAuthPasswordPaired tokensTailscale100.64.172.34100.122.189.81\nAccess\nVia Tailscale:\nssh openclaw@open-claw.kangaroo-eel.ts.net\nVia PVE:\nsshpass -p 'admin' ssh -o StrictHostKeyChecking=no root@192.168.178.39 "qm guest cmd 403 exec-status"\nRelated\n\nzeroclaw — Different daemon on CT333 (not the same!)\nnanobot — Also on CT333 (different from both)\nproxmox — PVE host\ntailscale — VPN routing\n"},"agents/qwenpaw":{"slug":"agents/qwenpaw","filePath":"agents/qwenpaw.md","title":"QwenPaw","links":["agents/hermes-production","systems/omniroute","systems/model-providers","infrastructure/proxmox"],"tags":["qwenpaw","ct465","agent","paseo"],"content":"QwenPaw\nPurpose\nMulti-agent gateway platform. Runs on CT465 with its own workspace, agent profiles, and model routing. Used for group chats and specialised agent tasks.\nLocation\n\nContainer: LXC CT465 (qwenpaw)\nHostname: qwenpaw\nSoftware: QwenPaw with Paseo daemon\n\nCurrent State\n\nStatus: Active\nDefault chat model: OmniRoute Smart Route (auto/smart)\nConfig format: JSON (agent.json per workspace)\nDaemon: paseo.service\n\nOperating Layers\nQwenPaw has several layers that must not be conflated:\n\nPVE/LXC layer: container status, guest access, resources\nPaseo layer: long-running daemon/supervisor (paseo.service)\nWorkspace layer: agent profiles with agent.json and active_model\nProvider layer: remote providers (Kilo/OpenRouter) or local (Ollama)\nChannel layer: Telegram or other messaging adapters\n\nKey Rules\n\nDo not assume “process is running” means “agent works.” Model resolution and a real response are separate checks.\nVerify model resolution per agent. A default agent can work while a specialist agent silently has no model.\nPreserve existing provider/model routes. Do not repin unrelated agents or replace known-good routes.\nDefault chat model: Use OmniRoute Smart Route (auto/smart) per Anthony’s preference.\n\nDiagnostics\nqwenpaw doctor --deep --timeout 10 --llm-timeout 20\nqwenpaw models list\nqwenpaw daemon status\nProvider Verification\nWhen adding an OpenAI-compatible provider from Proxmox:\n\nVerify target by VMID and hostname before writing config\nCT600/omniroute is the OmniRoute service at 100.88.81.19:20128\nCT601/omni is a separate guest at 100.93.204.4\nQuery /v1/models using the exact API key stored for QwenPaw, not a management key\n\nRelated\n\nhermes-production — Production Hermes\nomniroute — Model routing\nmodel-providers — Provider catalog\nproxmox — PVE host\n"},"agents/zeroclaw":{"slug":"agents/zeroclaw","filePath":"agents/zeroclaw.md","title":"ZeroClaw","links":["agents/nanobot","agents/open-claw","infrastructure/proxmox"],"tags":["zeroclaw","ct333","daemon","agent"],"content":"ZeroClaw\nPurpose\nAutonomous AI daemon running on CT333. Handles independent agent operations with TOML configuration. Runs alongside Nanobot on the same container but is completely separate software.\nLocation\n\nContainer: LXC CT333 (clawtest)\nTailscale IP: 100.122.189.81\nUser: root\nShares container with: nanobot\n\nCurrent State\n\nStatus: Active\nConfig format: TOML (/root/.zeroclaw/config.toml)\nDaemon binary: /root/.local/bin/zeroclaw daemon\nWorkspace: /root/.zeroclaw/workspace\nState: /root/.zeroclaw/state/\n\nKey Paths\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nItemPathConfig/root/.zeroclaw/config.tomlConfig dir/root/.zeroclaw/Daemon binary/root/.local/bin/zeroclawWorkspace/root/.zeroclaw/workspaceState/root/.zeroclaw/state/\nAccess\nsshpass -p 'admin' ssh -o StrictHostKeyChecking=no root@192.168.178.39 "pct exec 333 -- <command>"\nCritical Pitfalls\n\nDuplicate daemons break Telegram replies. If Telegram shows typing then no message, check for multiple daemon processes before changing providers.\nDo not confuse with Open-Claw. Open-Claw is VM 403, not CT333. Different config format (TOML vs JSON), different user (root vs openclaw), different auth (paired tokens vs password).\nDo not confuse with Nanobot. Nanobot is JSON config, ZeroClaw is TOML. Both on CT333 but completely separate.\n\nScope Discipline\n\nNo destructive operations without confirmation\nNo modifying running state without saving config first\nNo changes that could create lock-out scenarios without a safety net\n\nRelated\n\nnanobot — Different agent, same container\nopen-claw — Different VM, similar name\nproxmox — PVE host\n"},"comparisons/camoufox-vs-chrome":{"slug":"comparisons/camoufox-vs-chrome","filePath":"comparisons/camoufox-vs-chrome.md","title":"Camoufox vs Chrome","links":["systems/headless-server","decisions/","systems/browser-backend","concepts/browser-automation"],"tags":["comparison","browser"],"content":"Camoufox vs Chrome\nPurpose\nCompare the two browser-automation options for Hermes’s browser tool.\nAt a glance\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nCamoufoxChromeIn use on this installYes — configured cloud_providerNo evidence foundConfigbrowser.camofox.managed_persistence: true (note: key is spelled camofox in config, not camoufox)n/aDrivercua-driver v0.9.0 at /home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-drivern/aDisplayHeaded, via Xvfb :99 + fluxbox — see headless-servern/ause_gatewayfalsen/a\nDetail\nCamoufox (in use)\n\nVerified: browser.cloud_provider: camofox, inactivity_timeout: 120s, allow_private_urls: true.\nRuns headed under Xvfb rather than a true headless mode, per headless-server — fluxbox provides the window manager, x11vnc/noVNC expose it for human viewing.\ncua-driver bridges Hermes’s computer-use tool calls to the browser.\n\nChrome (not found in use)\n\nNo google-chrome, chromium, or CDP-only launch flags found in the sanitized config or running process list during this audit.\nIncluded in the comparison because the page existed as a placeholder — if Chrome/CDP was evaluated and rejected, that reasoning isn’t captured anywhere; worth a index entry if anyone remembers why Camoufox was chosen over stock Chrome.\n\nVerdict\nThis install uses Camoufox exclusively. There’s no evidence Chrome/Chromium was ever wired in as an alternative on this host — the comparison exists more as a placeholder for a decision that may have been made outside this wiki than as an active either/or choice today.\nRelated\n\nbrowser-backend\nbrowser-automation\n"},"comparisons/hermes-vs-nanobot":{"slug":"comparisons/hermes-vs-nanobot","filePath":"comparisons/hermes-vs-nanobot.md","title":"Hermes vs Nanobot","links":["systems/nanobot","systems/hermes-agent"],"tags":["comparison","hermes","nanobot"],"content":"Hermes vs Nanobot\nPurpose\nCompare Hermes (this agent) with Nanobot, another agent process in the homelab, and correct a stale claim in nanobot.\nCorrection to existing wiki content\nsystems/nanobot.md (written earlier the same day) states “No nanobot process running on this host” and “Unknown: whether nanobot should be redeployed on this host.” Both are superseded by direct verification during this pass:\n\nNanobot is running, just not on this host (ct460). It’s on LXC 333 (“clawtest”), Tailscale IP 100.122.189.81, process nanobot listening on port 8766.\nct333 also runs a whatsapp-bridge.service (“WhatsApp Bridge for Nanobot”) — a systemd service, active.\nIt’s publicly reachable via the same Cloudflare Tunnel that fronts Hermes: nanobot.martinwa.org → 100.122.189.81:8766, and nano-ssh.martinwa.org → SSH on the same host.\nThe earlier note “Nanobox on the PVE host at 192.168.178.39” conflated Nanobot with pve’s own LAN IP (that address is Proxmox’s web UI, per the tunnel’s pve.martinwa.org entry) — an error, not a second Nanobot instance.\n\nAt a glance\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nHermes (ct460)Nanobot (ct333 “clawtest”)StatusActive, primary agentActive, running (was previously assumed dormant)HostDedicated LXC, agent;ai;automation tagsLXC tagged/named “clawtest”Public exposurehermes.martinwa.org, hermes-desktop.martinwa.orgnanobot.martinwa.org, nano-ssh.martinwa.orgExtra servicesStratos command centre, browser/VNC stackWhatsApp bridgeRole todayPrimary personal agent (this wiki’s subject)Unclear — running, but purpose/overlap with Hermes not documented anywhere found\nVerdict\nBoth agents are live simultaneously, which the wiki didn’t previously reflect. Worth a real decision: is Nanobot still needed alongside Hermes, or is it a leftover that should be retired (freeing ct333) or given an explicit distinct role? This comparison surfaces the question rather than answering it — no evidence either way was found on either host.\nRelated\n\nnanobot — needs its “Current state” section updated to reflect this finding\nhermes-agent\n"},"comparisons/memory-backends":{"slug":"comparisons/memory-backends","filePath":"comparisons/memory-backends.md","title":"Memory Backends","links":["infrastructure/domains-and-tunnels","systems/mnemosyne","systems/memory-backend","concepts/agent-memory"],"tags":["comparison","memory"],"content":"Memory Backends\nPurpose\nDocument Hermes’s memory backend on this box, and record why the taxonomy this page assumed (Hindsight, Mnemosyne, file-backed) doesn’t actually apply here.\nCorrection (2026-07-22, per Anthony)\nThis wiki previously stated Hindsight was this box’s predecessor memory backend, based on archived hindsight-* script names found under ~/.hermes/scripts/ and a “Hindsight clean memory bank” mention in current-state.md. That was wrong — Hindsight has nothing to do with this Hermes box. It was confused with a different Hermes instance (most likely ct450 “hermesagent”, the older/original Hermes install still running in this homelab — see domains-and-tunnels). All Hindsight-specific pages (systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md) have been deleted rather than left as deprecated stubs, since they describe something that was never true of this box.\nThe three archived scripts (hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived) that had been sitting under ~/.hermes/scripts/ were removed on 2026-07-22 at Anthony’s request, closing out the mix-up entirely.\nWhat’s actually true here\n\nOnly backend ever verified on this box: Mnemosyne — local native plugin (~/.hermes/plugins/mnemosyne), not a docker service. See mnemosyne and memory-backend for full config.\nNo evidence of a file-backed-only memory mode being used either — it’s Mnemosyne or nothing on this install.\n\nOpenViking evaluation (2026-08-12) — NOT adopted\n\nAnthony explored OpenViking (Volcengine/ByteDance context database, filesystem-style knowledge hierarchy, tiered retrieval, 6-category auto extraction) as a potential memory provider.\nNot adopted: memory.provider is single-slot — setting hermes config set memory.provider openviking would fully replace Mnemosyne, with no parallel/dual-active mode.\nRisk assessment flagged: schema mismatch, embedding drift, scope leakage, provenance loss, and one-way/no-rollback migration with no dry-run snapshot.\nVerdict: stay on Mnemosyne. Config verified memory.provider: mnemosyne (2026-08-12). No migration was performed.\n\nRelated\n\nmnemosyne\nmemory-backend\nagent-memory\n"},"comparisons/model-routers":{"slug":"comparisons/model-routers","filePath":"comparisons/model-routers.md","title":"Model Routers","links":["decisions/keep-openrouter-as-fallback","systems/model-providers","systems/omniroute","systems/openrouter","concepts/model-routing"],"tags":["comparison","routing"],"content":"Model Routers\nPurpose\nCompare LiteLLM, OmniRoute, and OpenRouter as model-routing layers for Hermes.\nAt a glance\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nLiteLLMOmniRouteOpenRouterRole in HermesDefault/primary providerFallback/auxiliary providerNot configured as a standalone providerEndpointhttp://litellm:4000/v1http://omniroute:20128/v1 (public: omniroute.martinwa.org)n/aAdvertised model count~1512~3430n/aRouting styleStraight proxyCombo strategy, LKGP scoring, session affinityn/aAux roles used for—skills_hub, approval, mcp—Config presencecustom_providers in ~/.hermes/config.yamlsameabsent from custom_providers; only referenced via a kept keep-openrouter-as-fallback decision\nDetail\nLiteLLM — primary\nDefault provider for main chat traffic (freellmapi-auto default model resolves through it). See model-providers.\nFreeLLMAPI — the router behind freellmapi-auto\nUnified LLM router at 100.82.40.23:3001 (also freellmapi.martinwa.org). Verified 2026-08-12:\n\n3-axis weighted scoring: Reliability + Speed + Intelligence, with user-configurable weights.\nDynamic guardrail multiplier: Final score = weighted avg × (free-quota headroom × live rate-limit penalty). Below 1.0 means the model is held back.\nPenalty Inspector (“Router Pressure”): granular penalty factors (e.g. score × 0.5) from recent 4xx/5xx, cooldowns, RPM/RPD limits, monthly token caps.\nCapability-aware routing: filters candidates on whether the request has images (vision) or tool calls.\nExposes /api/fallback/routing, /api/fallback/penalty-inspector, /api/fallback/token-usage.\nThis is the router that resolves Hermes’s default model freellmapi-auto — distinct from OmniRoute’s LKGP (below).\n\nOmniRoute — fallback / auxiliary\nNot used for Hermes’s main chat path but handles specific auxiliary roles (skills_hub, approval, mcp) and serves as the fallback provider. Has a much larger advertised catalog (~3430 vs ~1512) but that catalog can diverge from what’s actually resolvable — combo discovery vs. resolution can differ (documented known issue). See omniroute.\nOpenRouter — kept only as a fallback decision, not wired in\nVerified: absent from custom_providers in the live config. A decision page (keep-openrouter-as-fallback) records intent to retain it as a fallback option, but there’s no live routing through it today — see openrouter for the “previously used, largely replaced” history.\nVerdict\nTwo routers are actually live (LiteLLM primary, OmniRoute fallback/auxiliary); OpenRouter is a retained intent, not a working path, as of 2026-07-22. If keep-openrouter-as-fallback is still desired, it needs an actual custom_providers entry — currently it’s fallback-in-name-only.\nRelated\n\nmodel-providers\nmodel-routing\n"},"comparisons/steel-vs-camoufox":{"slug":"comparisons/steel-vs-camoufox","filePath":"comparisons/steel-vs-camoufox.md","title":"Steel Browser vs Camoufox","links":["systems/browser-backend","concepts/browser-automation","decisions/rollback-steel-to-camofox"],"tags":["comparison","browser","steel","camofox","decision"],"content":"Steel Browser vs Camoufox\nPurpose\nCompare the two browser-automation backends available to Hermes, including the Jul 29–30 experiment and the decision to revert.\nAt a glance\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSteel Browser (CT201)Camoufox (CT450)In useDisabled since 2026-07-30Yes — primary backendEngineChromium 140 (Playwright)Camoufox (Firefox-based)ConnectionCDP (ws:// via port 9223)REST API (localhost:9093)APIREST at :3000 — create/destroy/query sessionsCamoufox REST + CDP WebSocketHeadlessNative headlessXvfb + VNC (headed, anti-fingerprint)ContainerDocker on CT201Systemd service on CT450StealthStandard ChromiumAnti-detection fingerprintingSession viewerBuilt-in UI at /steelVNC at /vnc-camofoxPluginbrowser-steel plugin (disabled)Native Hermes browser.cloud_provider: camofox\nThe Steel Experiment (Jul 29–30)\nWhy it was tried\n\nSteel promised full CDP access for richer browser control\nBuilt-in session viewer and management UI\nDocker-based, clean separation from Hermes host\n\nTimeline\n\nJul 29: Steel deployed on CT201 via Docker Compose, Chromium 140. Hermes configured with browser.cloud_provider: steel, browser-steel plugin enabled.\nJul 29 (later): CDP connectivity issues surfaced — Hermes was reaching 0.0.0.0 from the plugin’s internal CDP URL. Required _normalize_cdp_url() patch in the Steel plugin to remap to 100.96.244.39.\nJul 30: DeepSeek V4 Flash 0731 went live and rejected tool names that didn’t match ^[a-zA-Z0-9_-]+$. The Steel plugin’s tools (steel_scrape, steel_session_options) were fine, but the route through OmniRoute → Steel → CDP was unreliable with empty-stream errors.\nJul 30 (evening): Decision to revert — Camoufox re-enabled, browser-steel plugin disabled, CAMOFOX_URL and CAMOFOX_API_KEY uncommented in .env. Steel left running on CT201 but Hermes no longer routes through it.\n\nWhat broke\n\nCDP URL normalization required a plugin patch to handle Docker’s internal networking\nEmpty-stream errors from opencode-go/deepseek-v4-flash provider path — the combo strategy does NOT iterate on empty responses, they pass to Hermes’s own fallback chain (the #1 cause of “combo works but Hermes falls back”)\nTool name validation on DeepSeek 0731 was stricter — this affected all providers, not just Steel\n\nLesson learned\nSteel is a capable backend but adds a network hop and another plugin layer. Camoufox runs locally on CT450 with fewer failure modes. The browser-steel plugin still exists (disabled) if needed in future.\nDetail\nCamoufox (current primary)\n\nDeployed: Persistent systemd service on CT450\nConnection: REST API at localhost:9093, profile anthony, sessionKey anthony\nKey advantage: Runs on the same host as Hermes — no network hop, no container networking to debug\nAuth: Already authenticated into services (Uber Eats, LinkedIn, etc.) — no cookie files needed\nLimitation: Headed via Xvfb, consumes display resources\n\nSteel Browser (disabled but available)\n\nDeployed: 2026-07-29 on CT201, Docker, custom steel-browser-api:fixed image\nStatus: Still running on CT201, Hermes not connected\nCDP endpoint: http://100.96.244.39:9223\nREST API: http://100.96.244.39:3000\nUI: runtipi.kangaroo-eel.ts.net/steel\nPlugin: browser-steel at /home/hermes/.hermes/plugins/browser-steel/ — disabled, plugin_steel toolset flagged requires_health=steel\n\nRelated\n\nbrowser-backend\nbrowser-automation\nrollback-steel-to-camofox\n"},"concepts/agent-memory":{"slug":"concepts/agent-memory","filePath":"concepts/agent-memory.md","title":"Agent Memory","links":["systems/mnemosyne","comparisons/memory-backends","systems/memory-backend"],"tags":["concept","memory"],"content":"Agent Memory\nIdea\nHow an agent retains facts/context beyond a single conversation turn or session — what gets stored, when it’s flushed, and how it’s retrieved later.\nHow this install implements it\n\nBackend: Mnemosyne, local native (not a separate docker service) — see mnemosyne, memory-backends.\nVerified settings: memory enabled true; message char limit 5000; user-profile char limit 3000; flush every 6 turns; auto_sleep: true; default scope session; vector type int8; shared surface data/shared/mnemosyne.db.\nCorrection (2026-07-22, per Anthony): this page previously named Hindsight as a predecessor backend on this box. That was a mix-up with a different Hermes instance — Hindsight has nothing to do with this box. See memory-backends for the full correction.\n\nModel\n\nPer-turn conversation content accumulates up to the char limit, then flushes to the Mnemosyne store every 6 turns (rather than every turn) — a batching tradeoff between write frequency and losing very recent context if the process dies mid-batch.\nscope: session means recall defaults to the current session rather than a global cross-session pool, unless explicitly widened.\n\nRelated\n\nmemory-backend\nmemory-backends\n"},"concepts/browser-automation":{"slug":"concepts/browser-automation","filePath":"concepts/browser-automation.md","title":"Browser Automation","links":["comparisons/camoufox-vs-chrome","systems/headless-server","comparisons/steel-vs-camoufox","decisions/rollback-steel-to-camofox","systems/browser-backend","concepts/tool-calling"],"tags":["concept","browser","camofox","steel"],"content":"Browser Automation\nIdea\nGiving an agent a real, controllable web browser — as opposed to just an HTTP-fetch tool — so it can handle JS-heavy pages, logins, and visual verification.\nHow this install implements it\n\nEngine: Camoufox (Firefox-based, fingerprint-resistant) — see camoufox-vs-chrome for why this over stock Chrome.\nConnection: REST API at localhost:9093 (profile anthony) + CDP WebSocket at localhost:9377\nDisplay: headed, not truly headless — runs under Xvfb :99 (1600×1200×24) with fluxbox as window manager, viewable live over VNC (see headless-server).\nConfig: cloud_provider: camofox, camera_provider: camofox in ~/.hermes/config.yaml\nAuth: Persistent profile anthony — authenticated into Uber Eats, LinkedIn, etc. No cookie files needed.\n\nWhy headed-under-Xvfb instead of headless\nRunning a real X display (rather than a headless browser flag) lets a human watch/intervene via VNC and tends to reduce automation-detection fingerprints compared to headless-mode flags — consistent with using Camoufox in the first place. The tradeoff, flagged in headless-server, is that the VNC exposing this display currently has no password.\nThe Steel Experiment (Jul 29–30)\nSteel Browser (Chromium 140 on CT201) was briefly the primary backend. It offered full CDP access and a built-in session viewer, but was reverted after:\n\nCDP URL normalization issues (Docker internal networking required a plugin patch)\nEmpty-stream errors from the opencode-go/deepseek-v4-flash provider path\nDeepSeek 0731’s stricter tool name validation affected all providers\n\nLesson: A local browser backend (same host, no network hop) has fewer failure modes than a remote Docker container. The browser-steel plugin is still present but disabled. See steel-vs-camoufox and rollback-steel-to-camofox.\nRelated\n\nbrowser-backend\nsteel-vs-camoufox\ncamoufox-vs-chrome\nheadless-server\ntool-calling\n"},"concepts/model-routing":{"slug":"concepts/model-routing","filePath":"concepts/model-routing.md","title":"Model Routing","links":["systems/model-providers","systems/omniroute","systems/openrouter"],"tags":["routing","model","inference"],"content":"Model Routing\nPurpose\nHow Hermes selects providers, applies combos, and enforces context limits.\nCurrent Flow\n\nMain path: custom:litellm → http://litellm:4000/v1\nDefault model: freellmapi-auto\nOmniRoute used for auxiliary roles: vision, skills_hub, approval, mcp, title_generation, web_extract, compression\n\nContext Enforcement\n\nThree layers: model native → combo data.config.context_length → Hermes client cap\nEffective context is the minimum of the three\n\nCombo Behavior\n\npriority retries same provider up to maxRetries, then errors; it does NOT fall through\nfill-first walks the list on failure\nauto/lkgp recalculates per request using health scoring\nSame-provider entries in a combo share rate-limit buckets\nSession affinity caches provider in session_model_history\n\nAliases and Mappings\n\nModel aliases resolved via key_value table (modelAliases namespace)\nWildcard mappings via model_combo_mappings\nOpenCode/Zen entries may resolve back to opencode-zen provider internally\n\nHistory\n\nUser-provided: OpenRouter was used previously; provider lists now centered on LiteLLM + OmniRoute\nUser-provided: persist_switch_by_default=false; /model is per-session unless --global used\nInferred: Configuration undergoes periodic corrections related to vision, model fallback, and browser backend tuning\n\nRelated\n\nmodel-providers\nomniroute\nopenrouter\n"},"concepts/persistent-shells":{"slug":"concepts/persistent-shells","filePath":"concepts/persistent-shells.md","title":"Persistent Shells","links":["runbooks/troubleshoot-tool-loop","systems/terminal-backend","systems/hermes-agent","concepts/tool-calling"],"tags":["concept","terminal"],"content":"Persistent Shells\nIdea\nA background shell session that survives across separate Hermes turns/requests, instead of spawning and tearing down a fresh shell each time — so state (working directory, exported variables, long-running processes) carries over.\nHow this install implements it\n\nVerified (~/.hermes/config.yaml): terminal.persistent_shell: true\nBackend: local (not containerized separately from the agent process)\nWorking directory: /home/hermes/workspace\nTimeout: 180s\nHome mode: auto\nContainer CPU: 1\n\nWhy it matters here\nEnables multi-step CLI workflows (e.g. a long git operation, a build, a watch process) to be resumed across turns without Hermes re-establishing context each time. The tradeoff is a shell can be left in a bad state (stuck process, wrong directory) that silently affects the next turn — see troubleshoot-tool-loop for the related tool-loop guardrails that catch some of this.\nRelated\n\nterminal-backend\nhermes-agent\ntool-calling\n"},"concepts/tool-calling":{"slug":"concepts/tool-calling","filePath":"concepts/tool-calling.md","title":"Tool Calling","links":["runbooks/troubleshoot-tool-loop","concepts/persistent-shells","systems/hermes-agent"],"tags":["concept","tool-calling"],"content":"Tool Calling\nIdea\nHow Hermes decides to invoke a tool, and — more specifically for this page — how it detects and stops itself when tool use goes wrong (loops, repeated failures).\nHow this install implements guardrails\nVerified (~/.hermes/config.yaml, tool_loop_guardrails:):\nwarnings_enabled: true\nhard_stop_enabled: false\nwarn_after:\n  exact_failure: 2\n  same_tool_failure: 3\n  idempotent_no_progress: 2\nhard_stop_after:\n  exact_failure: 5\n  same_tool_failure: 8\n  idempotent_no_progress: 5\n\n\nWarnings trigger well before any hard stop (2–3 repeats vs 5–8).\nhard_stop_enabled: false — Hermes will warn about a stuck tool loop but will not currently force-stop itself. That’s a deliberate-looking but consequential setting: a genuinely stuck loop only gets a warning, not an automatic halt.\n\nWhy it matters here\nThis is the mechanism behind troubleshoot-tool-loop — that runbook’s “agent loops on one tool” symptom is exactly what warn_after/hard_stop_after are meant to catch. With hard-stop disabled, a human (or the messaging integrations) is the actual backstop today.\nRelated\n\npersistent-shells\ntroubleshoot-tool-loop\nhermes-agent\n"},"current-state":{"slug":"current-state","filePath":"current-state.md","title":"Current State","links":["comparisons/steel-vs-camoufox","decisions/rollback-steel-to-camofox","incidents/2026-08-01-mnemosyne-capture-failure","incidents/2026-08-04-omniroute-free-deepseek-silent-stops","systems/omniroute","systems/nanobot","systems/homepage-dashboard","systems/scheduled-tasks","systems/guanaco","systems/model-providers","systems/browser-backend","systems/mnemosyne"],"tags":["system"],"content":"Current State\nUpdated: 2026-08-14\nAttested facts\n\nConfig version: 33 — Verified, ~/.hermes/config.yaml, 2026-07-25\nDefault profile: default — Verified, WebUI prompt, 2026-07-22\nTimezone: Australia/Perth — User-provided + Verified in config, 2026-07-22\nMain provider: custom:litellm → http://litellm:4000/v1 — Verified, ~/.hermes/config.yaml, 2026-07-22\nDefault model: freellmapi-auto — Verified, ~/.hermes/config.yaml, 2026-07-22\nFallback/aux provider: custom:omni → http://omni:20128/v1 — Verified, config, 2026-08-13 (migrated from custom:omniroute, CT600)\nOmniRoute host: omni.kangaroo-eel.ts.net → 100.93.204.4 — Verified, DNS resolution, 2026-08-13\nMemory backend: Mnemosyne — Verified, config, 2026-07-22\nMemory enabled: true — Verified, config, 2026-07-22\nStreaming enabled: true — Verified, config, 2026-07-22\nDashboard public URL: hermes.martinwa.org/ — Verified, config, 2026-07-22\nPet config: enabled/slug null — Verified, config, 2026-07-22\nWebUI connected platforms: local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22\n\nRecent Changes (2026-08-02)\n\nBrowser backend reverted: Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. browser.cloud_provider: camofox, camera_provider: camofox. Steel plugin disabled. See steel-vs-camoufox and rollback-steel-to-camofox.\nMnemosyne capture fix (2026-08-01): Automatic conversational capture was broken by stringified YAML (sync_roles: "['user']"). Fixed by rewriting structured Mnemosyne settings with native YAML types. See 2026-08-01-mnemosyne-capture-failure.\nMnemosyne vector coverage gap discovered: Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.\nDeepSeek V4 Flash 0731: Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (^[a-zA-Z0-9_-]+$) — may cause issues with OmniRoute-transformed tool names.\nUber Eats report upgraded: Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with camofox-browser-automation + browser-backend-verification skills.\nGitHub Trending Radar — Daily Digest removed: Stopped by user on 2026-08-02. Weekly roundup still active.\nOps profile renamed: Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.\nWiki audit cron job created: Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.\nOps profile Mnemosyne enabled: Mnemosyne plugin enabled on ops profile with profile_isolation: true — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: ~/.hermes/profiles/ops/config.yaml (memory.mnemosyne.profile_isolation: true, allow_tool_override: false).\nOmniRoute free-DeepSeek silent-stop mitigation (2026-08-04): Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See 2026-08-04-omniroute-free-deepseek-silent-stops.\nOmniRoute three-layer fix (2026-08-07): npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See omniroute.\nNanobot status degraded (2026-08-07): CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See nanobot.\nCT460 memory bumped (2026-08-07): 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.\n\nRecent Changes (2026-08-08)\n\nCron job model repinning: 3 jobs repinned from dead freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker repinned from dead freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named cron jobs (e.g. vertex/gemini-2.5-flash, gpt-5.6-terra, deepseek-v4-flash-free) repinned to smart-route. Daily Marketing Job Radar swapped to gpt-5.6-terra.\nHermes UI vite config revert: Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build now serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.\nPhase One estate review completed: Manual cron run verified CT450’s 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460’s 11 production jobs remain enabled with last_status=ok, and CT333 Nanobot has a live jobs.json with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.\n\nRecent Changes (2026-08-09)\n\nPortainer password reset (CT245): Admin user is anthony (not admin). Password was forgotten; reset via docker run --rm -v portainer_data:/data portainer/helper-reset-password after stopping the Portainer container. Verified end-to-end with POST /api/auth → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let’s Encrypt cert on :9443 (see homepage-dashboard). Recovery procedure: stop container → run helper image → restart → verify with auth API.\nSFTPGo security review (CT299): Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path // resolves to / — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is root, (4) quota value F1 on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony’s review.\n\nRecent Changes (2026-08-10)\n\nCron job failures — session storage issue: Four cron jobs failed on 2026-08-10 with two distinct error patterns:\n\nTimeoutError (idle for 601–602s): “Daily Marketing Job Radar” (dca8482e4f76) at 07:12, “Political News Digest (Mon/Thu)” (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.\nRuntimeError (session storage): “Tech & AI Newsletter Digest (Analytical)” (85edf6772bca) at 07:12, “GitHub Trending Radar — Weekly Roundup” (c0a9fa314542) at 09:03. Error: “session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions).”\nRoot cause: Disk at 83% (11GB free on 65GB). state.db is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.\nImpact: News digests and GitHub Trending did not deliver. Job Radar did not run.\nSee: scheduled-tasks Known Issues section.\n\n\nOpenCode WebSocket proxy errors: Repeated “OpenCode service unavailable” errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, opencode serve), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.\n\nRecent Changes (2026-08-14)\n\nOmniRoute factory reset on CT601: Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at /opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See omniroute.\nHermes default model changed: auto/best-free → resilient-free in ~/.hermes/config.yaml. auto/best-free was misconfigured (no combo definition, routed to paid gpt-5.6-sol).\nMultiple free combos broken by upstream model ID changes: OpenRouter Poolside laguna-s-2-1:free → laguna-s-2.1:free (dash→dot); Groq llama-4-scout-17b-16e-instruct retired. Affected: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free.\nGuanaco router updated: CT205 ollama container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See guanaco.\nGPT-5.6 Luna research: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See model-providers.\nHermes UI restart: Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.\nLiteLLM routing issue identified: Hermes was previously pointed at LiteLLM (http://litellm:4000/v1, model free-auto) causing 6–26s per call. OmniRoute auto/best-free tested at 265ms. Config now points at OmniRoute.\n\nRecent Changes (2026-08-13)\n\nOmniRoute migrated CT600 → CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Backup: ~/.hermes/config.yaml.bak.omniroute-removal. See omniroute.\nEvening stand-down briefing weather bugfix: Weather collector returns a plain string (+16°C Sunny), but the three dashboard widgets (evening.html, evening-grid.html, evening-briefing.html) read object fields → undefined. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.\n\nRelated\n\nbrowser-backend\nmnemosyne\nscheduled-tasks\nsteel-vs-camoufox\nrollback-steel-to-camofox\n"},"decisions/index":{"slug":"decisions/index","filePath":"decisions/index.md","title":"Decisions","links":["decisions/keep-openrouter-as-fallback","decisions/persistent-shell-enabled","decisions/rollback-steel-to-camofox","current-state","concepts/model-routing"],"tags":["decisions"],"content":"Decisions\nStable strategic and config decisions, with evidence and status.\nCurrent decisions\n\nkeep-openrouter-as-fallback\npersistent-shell-enabled\nrollback-steel-to-camofox\n\nHistory\n\nVerified: Decision pages initialized during wiki setup on 2026-07-22\nInferred: Config version 33 and multiple named backups imply iterative changes in vision, browser, and model routing\nUnknown: Exact date each listed decision was made; exact alternatives considered\n\nRelated\n\ncurrent-state\nmodel-routing\n"},"decisions/keep-openrouter-as-fallback":{"slug":"decisions/keep-openrouter-as-fallback","filePath":"decisions/keep-openrouter-as-fallback.md","title":"Keep OpenRouter as fallback","links":["concepts/model-routing","systems/omniroute","systems/openrouter"],"tags":["routing","provider","decision"],"content":"Keep OpenRouter as fallback\nDecision\nOpenRouter must remain available as a fallback provider, even if a self-hosted router becomes primary.\nReasons\n\nFree models are important.\nBroad model coverage.\nPrevious migrations disrupted provider availability.\nA fallback reduces the chance of Hermes becoming unusable.\n\nAlternatives considered\n\nDirect providers only\nOmniRoute exclusively\nLocal models exclusively\n\nRevisit when\nA self-hosted router has operated reliably for at least 30 days.\nRelated\n\nmodel-routing\nomniroute\nopenrouter\n"},"decisions/persistent-shell-enabled":{"slug":"decisions/persistent-shell-enabled","filePath":"decisions/persistent-shell-enabled.md","title":"Persistent shell enabled","links":["concepts/persistent-shells"],"tags":["automation","shell","decision"],"content":"Persistent shell enabled\nDecision\nEnable persistent shells for Hermes.\nReasons\n\nPreserve terminal state across turns\nAvoid re-activating virtualenvs and aliases every call\n\nAlternatives considered\n\nAlways transient shell\nManual re-activation by user\n\nRevisit when\nUse case becomes critical again.\nRelated\n\npersistent-shells\n"},"decisions/rollback-steel-to-camofox":{"slug":"decisions/rollback-steel-to-camofox","filePath":"decisions/rollback-steel-to-camofox.md","title":"Rollback Steel to Camoufox","links":["comparisons/steel-vs-camoufox","systems/browser-backend","concepts/browser-automation"],"tags":["decision","browser","steel","camofox","rollback"],"content":"Decision: Rollback Steel Browser → Camoufox\nDate\n2026-07-30\nContext\nSteel Browser was deployed on CT201 on 2026-07-29 as the primary browser automation backend, replacing Camoufox. It ran for approximately 24 hours before being reverted.\nOptions Considered\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nOptionProsConsKeep SteelFull CDP access, session viewer UI, Docker isolationCDP URL normalization issues, network hop, empty-stream errorsRevert to CamoufoxLocal (no network hop), already authenticated, fewer failure modesHeaded display consumes resources, no built-in session viewerBoth (Steel primary, Camoufox fallback)RedundancyComplexity, two backends to maintain, unclear which one Hermes uses per-request\nDecision\nRevert to Camoufox as primary. Steel plugin disabled but left running on CT201.\nRationale\n\nReliability: Camoufox runs on the same host as Hermes (CT450) — no Docker networking, no CDP URL normalization patches, no container-to-host routing issues.\nAuth persistence: Camoufox’s persistent profile anthony is already authenticated into key services (Uber Eats, LinkedIn). Steel required cookie re-exports.\nFailure mode: The opencode-go/deepseek-v4-flash empty-stream errors were hitting all providers, but Steel added an extra failure surface (CDP connection, plugin layer) on top of the same problem.\nDeepSeek 0731 compatibility: The new model’s stricter tool name validation (^[a-zA-Z0-9_-]+$) affected all providers — not a Steel-specific issue, but another reason to minimise the toolchain.\n\nConsequences\n\nPositive: Browser automation is stable again. Uber Eats report ran successfully via Camofox on 2026-08-01.\nNegative: Steel’s session viewer UI and full CDP access are no longer available to Hermes.\nNeutral: Steel still runs on CT201 — can be re-enabled by uncommenting .env lines and enabling the plugin.\n\nEvidence\n\nCamoufox verified working Jul 30: browserConnected: true, sessions: 1, tabs: 1, active 200s\nUber Eats report (Aug 1) successfully scraped via Camofox REST API — 11 orders, $264.51\nSteel plugin browser-steel still present at /home/hermes/.hermes/plugins/browser-steel/ — disabled\n\nRelated\n\nsteel-vs-camoufox\nbrowser-backend\nbrowser-automation\n"},"experiments/active/example":{"slug":"experiments/active/example","filePath":"experiments/active/example.md","title":"Example experiment","links":[],"tags":["experiment"],"content":"Example experiment\nQuestion\nBaseline\nConfiguration tested\nResults\nVerdict\nPending.\nNext action"},"experiments/index":{"slug":"experiments/index","filePath":"experiments/index.md","title":"Experiments","links":["runbooks/update-hermes-safely","current-state"],"tags":["experiments"],"content":"Experiments\nNon-canonical tests, tuning passes, and change attempts.\nKnown experiments\n\npre-vision-fix-20260718-195712 config snapshot — Verified, backup file exists, 2026-07-18\npre-model-fix config snapshot — Verified, backup file exists\npre-camofox-tuning config snapshot — Verified, backup file exists, 2026-07-20\n\nOutcomes\n\nVerified: Backups exist with the above names\nUnknown: Whether each experiment succeeded, failed, or was reverted\nExcluded: No secrets, tokens, or full config dumps are stored here\n\nHistory\n\nVerified: Three named pre-change backups present as of 2026-07-22\nInferred: Backups were taken before optional config changes and left intact\n\nRelated\n\nupdate-hermes-safely\ncurrent-state\n"},"incidents/2026-07-29-mnemosyne-relay":{"slug":"incidents/2026-07-29-mnemosyne-relay","filePath":"incidents/2026-07-29-mnemosyne-relay.md","title":"2026-07-29-mnemosyne-relay","links":[],"tags":[],"content":"mnemosyne-tools-none — 2026-07-29\nFIXED: memory.mnemosyne.tools: None in config.yaml\nRoot cause: ~/.hermes/config.yaml had tools: None under the mnemosyne section (line 183). The memory manager interprets None as a literal tool name filter and fails schema loading with:\nWARNING: Memory provider 'mnemosyne' get_tool_schemas() failed: Unknown Mnemosyne tool(s) in memory.mnemosyne.tools: None.\n\nFallback: the next line registers all 40 tools anyway — no data loss, just noise.\nFix applied: Removed the tools: None line. Config now defaults to all tools (correct behaviour).\nTimeline: 86 occurrences logged between Jul 28–29, 19 unique sessions affected.\n\nrelay-scope-corruption — 2026-07-29\nSYMPTOM\nWARNING agent.relay_runtime: Hermes Relay turn finalization failed\nRuntimeError: invalid argument: scope handle is not at the top of the stack\n  File relay_runtime.py:644  end_turn → run_in_session → invoke\n  File nemo_relay/scope.py:144  pop()\n\nDETAILS\n\nHermes Agent 0.19.0, nemo_relay 0.6.0\n3 occurrences across 3 sessions within 33 min (07:37–08:10 UTC 2026-07-29)\nSessions: 20260723_024537_95c0e6f7, 20260728_233758_0c6b3c0d, 20260729_070054_b8452262\nException is caught (WARNING, not ERROR). Turn finalization may be incomplete.\nNo recurrence after 08:10.\n\nLIKELY CAUSE\nScope push/pop ordering mismatch. Something pushes a scope that isn’t at the top when end_turn tries to pop it. Could be a recovery code path (tool failure → cleanup) pushing scopes in wrong order, or a subagent spawning pattern leaking a scope.\nREPORTED\nFiled (or should be filed) under NousResearch/hermes-agent issues. No token available to auto-create.\nMONITORING\nWatch for recurrence. If persists, check whether the affected sessions share a common tool call pattern (subagent delegation, tool failure, etc.)."},"incidents/2026-08-01-mnemosyne-capture-failure":{"slug":"incidents/2026-08-01-mnemosyne-capture-failure","filePath":"incidents/2026-08-01-mnemosyne-capture-failure.md","title":"Mnemosyne Automatic Capture Failure","links":["systems/mnemosyne","systems/memory-backend","incidents/2026-07-29-mnemosyne-relay"],"tags":["memory","mnemosyne","hermes","incident"],"content":"Mnemosyne Automatic Capture Failure — 2026-08-01\nImpact\nOrdinary Hermes conversations were not being written to Mnemosyne working_memory. Explicit memory tools, SQLite integrity, vectors, FTS indexes, consolidation, backups, and cron monitoring remained operational. The newest automatic/explicit working row before repair was approximately fourteen hours old despite active Hermes sessions.\nThe always-injected legacy files were also effectively full: MEMORY.md was 5,055 bytes against a 5,000-character limit and USER.md was 2,958 bytes against 3,000. Legacy writes had begun failing.\nRoot Cause\nStructured values under memory.mnemosyne in ~/.hermes/config.yaml had been serialized as strings instead of native YAML types. The critical value was:\nsync_roles: "['user']"\nThe provider interpreted this as the literal invalid role ['user'], so turn synchronization completed without persisting a user row. Other malformed values included skip_contexts, reflect, sleep_threshold, and ignore_patterns.\nRepair\n\nBacked up config to /home/hermes/.hermes/backups/mnemosyne-capture-fix-20260801_180249/config.yaml.before.\nRewrote structured Mnemosyne settings with native YAML types.\nSet profile_isolation: true because the installation has multiple Hermes profiles.\nPreserved sync_roles: [user] and the intended skipped contexts.\nRan hermes config check successfully.\nCreated authoritative Mnemosyne canonical slots for user preferences and key workflows, correcting stale light-theme, numbered-Obsidian, and CT460 route entries.\nBacked up legacy files to /home/hermes/.hermes/backups/legacy-memory-slim-20260801_180702/.\nReduced legacy injection from 8,013 bytes to 1,080 bytes while retaining essential always-on rules.\n\nVerification\nA pre-fix CLI turn using marker MNEMO_E2E_20260801_180113_27053 completed but produced no database row.\nA post-fix CLI turn using marker MNEMO_FIXED_20260801_180310_16583 wrote:\n[USER] Reply with exactly OK. Memory capture test marker: MNEMO_FIXED_20260801_180310_16583\nThe row was verified directly in ~/.hermes/mnemosyne/data/mnemosyne.db. Final diagnostics reported zero failed checks, complete working-vector coverage, zero orphan embeddings, 1,482 working rows, and 463 episodic vectors.\nRemaining Activation Step\nThe already-running Telegram gateway/provider instance was initialized before the config repair. Run hermes gateway restart from an external terminal to load the corrected typed configuration for messaging-platform turns. The gateway cannot safely restart itself from one of its own child tool calls.\nMonitoring Note\nThe health script checks capture staleness only after 24 hours. It therefore remained green during this fourteen-hour active-capture outage. A future improvement should compare recent non-cron conversation activity with the newest working_memory row rather than lowering the threshold blindly and creating idle-period false positives.\nRelated\n\nmnemosyne\nmemory-backend\n2026-07-29-mnemosyne-relay\n"},"incidents/2026-08-04-omniroute-free-deepseek-silent-stops":{"slug":"incidents/2026-08-04-omniroute-free-deepseek-silent-stops","filePath":"incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md","title":"2026-08-04-omniroute-free-deepseek-silent-stops","links":["systems/omniroute"],"tags":[],"content":"[2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)\nStatus\nMitigated — combos reworked; upstream still degraded.\nSymptom\nOn both Hermes instances (ops/Tusk + the other instance), turns using deepseek-flash-free (and smart-route via good-free) did one tool call, then went silently — no text, no error. “continue” → one more tool call → silent stop again.\nRoot cause\nThe free-DeepSeek upstreams behind the combos started returning valid-but-empty HTTP 200s right after a tool_calls turn (no content, no further tool call). Hermes can’t distinguish “empty” from “model chose to stop” → ends turn silently. Began 2026-08-01 ~09:14 UTC, still active. OmniRoute log markers:\n[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)\n[ProxyEgress] <provider>/<conn-hash> ... status=error\n\nPer-model evidence (counts in storage.sqlite era log):\n\n238d5978/deepseek-v4-flash — 63× empty-after-tool (conn d4b9fee0, 189 proxy errors)\n238d5978/deepseek-v4-flash-0731 — 70× empty-after-tool\neb5af0da/deepseek-v4-flash-thinking:free (conn f8169018) — 504 proxy errors (combo #1 slot)\nnvidia deepseek + opencode-zen deepseek-free — noisy, errors\n\nStrategy trap: both combos used priority, which retries the same broken connection and never fails over on an empty 200.\nChange applied (2026-08-04, ~03:30 AWST)\nEdited combos in /var/lib/omniroute/storage.sqlite on CT600 (live DB, hot-reloads — no restart):\n\ndeepseek-flash-free — strategy priority → fill-first; 8 → 7 models: healthy free first (mimo-v2.5-free, laguna-s-2.1:free ×2 providers, step-3.7-flash, glm-5.2-cheap:free), one deepseek (deepseek-v4-flash-thinking:free) demoted to last resort.\ngood-free — strategy priority → fill-first; 15 → 12 models. Dropped the proven culprits: 238d5978/deepseek-v4-flash, nvidia/deepseek-ai/deepseek-v4-pro, nvidia/deepseek-ai/deepseek-v4-flash. opencode/deepseek-v4-flash-free kept as last resort.\n\nBackup of both pre-change rows: /tmp/combo_backup_20260804.json on the Hermes gateway LXC.\nVerification\n\nLive API (/v1/combos) reflects both changes immediately (hot reload).\n2-step tool-call round-trip through deepseek-flash-free returns proper final content.\nDirect round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.\n\nFollow-ups\n\nUpstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.\nConsider auto+explorationRate for self-tuning if free tier stays flaky.\nHermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface “(empty after tool calls)” more visibly.\n\nRefs\n\nomniroute\n"},"index":{"slug":"index","filePath":"index.md","title":"index","links":["current-state","agents/","systems/hermes-agent","systems/current-profile","agents/hermes-production","agents/hermes-chaos","agents/nanobot","agents/qwenpaw","agents/zeroclaw","agents/open-claw","systems/guanaco","systems/model-providers","systems/memory-backend","systems/mnemosyne","systems/omniroute","systems/openrouter","systems/opencode-go","systems/nanobot","systems/browser-backend","systems/hermes-ui","systems/terminal-backend","systems/tool-search","systems/messaging-integrations","systems/scheduled-tasks","systems/skills-index","systems/homepage-dashboard","systems/home-assistant","reports/uber-eats/","systems/obsidian","infrastructure/hosts","infrastructure/docker-services","infrastructure/credentials-map","infrastructure/network-map","infrastructure/domains-and-tunnels","infrastructure/backups","infrastructure/proxmox","infrastructure/tailscale","systems/cloudflare-tunnel","concepts/agent-memory","concepts/model-routing","concepts/persistent-shells","concepts/browser-automation","concepts/tool-calling","decisions/","decisions/keep-openrouter-as-fallback","decisions/persistent-shell-enabled","comparisons/camoufox-vs-chrome","comparisons/hermes-vs-nanobot","comparisons/memory-backends","comparisons/model-routers","runbooks/multiplexer-setup","runbooks/diagnose-docker","runbooks/recover-docker-service","runbooks/restart-hermes","runbooks/restart-browser","runbooks/update-hermes-safely","runbooks/restore-openrouter","runbooks/provider-health","runbooks/troubleshoot-tool-loop","runbooks/gateway-resource-resilience","runbooks/job-radar","raw/configs/hermes-config-sanitized.txt"],"tags":[],"content":"Anthony’s Systems Wiki\n\nOperational knowledge for Anthony’s agents, infrastructure,\nmodel routing and automation. Last updated: 2026-08-14\n\nStart here\n\ncurrent-state — Current systems status\nindex — Agent estate map (start here for “what runs where”)\nhermes-agent — Hermes Agent overview and version\ncurrent-profile — Active Hermes profile details\n\nAgent Estate\n\nindex — Estate overview and relationships\nhermes-production — CT460 production Command Centre (Rhino)\nhermes-chaos — CT450 experimental Chaos Lab\nnanobot — CT333 Scout (quota notifications, daily briefing)\nqwenpaw — CT465 multi-agent gateway\nzeroclaw — CT333 autonomous daemon (TOML config)\nopen-claw — VM 403 autonomous gateway node\n\nSystems\nCore\n\nhermes-agent — Agent version, install, config paths\ncurrent-profile — Active profile, default settings\nguanaco — Guanaco LLM proxy on CT205 (Ollama stack)\nmodel-providers — LiteLLM + OmniRoute providers, routing setup\nmemory-backend — Mnemosyne vector memory, settings\nmnemosyne — Mnemosyne cron jobs and config evidence\nomniroute — OmniRoute catalog, combos, and host path\nopenrouter — OpenRouter current state and history\nopencode-go — OpenCode Go subscription, model limits, pricing\nnanobot — Nanobot current state and history\nbrowser-backend — Camoufox (primary) + Steel (disabled), CDP config\nhermes-ui — Browser Hermes Desktop (przbadu/hermes-ui)\nterminal-backend — Shell persistence, working directory\ntool-search — Always-on lazy discovery for external tools\n\nIntegrations\n\nmessaging-integrations — Telegram, Discord, email, Slack, etc.\nscheduled-tasks — 28 active cron jobs\nskills-index — Installed skills inventory\n\nMonitoring & Dashboards\n\nhomepage-dashboard — Homepage dashboard (CT201)\nhome-assistant — Home Assistant integration\n\nReports\n\nindex — Uber Eats monthly spend reports\nobsidian — Obsidian vault structure and conventions\n\nInfrastructure\nServices\n\nhosts — Hosts, containers, important paths\ndocker-services — Docker compose files\ncredentials-map — Where credentials live (no values)\nnetwork-map — Zones, hosts, and ports across LAN, Tailscale, and the public tunnel\ndomains-and-tunnels — martinwa.org hostnames and Cloudflare Tunnel routing\nbackups — hermes backup CLI, retention, and gaps\n\nNetworking\n\nproxmox — PVE host, VMs, LXC containers\ntailscale — Tailscale serve routes and VPN\ncloudflare-tunnel — ct207-hosted tunnel serving Hermes publicly\n\nConcepts\n\nagent-memory — Memory model\nmodel-routing — Routing concepts\npersistent-shells — Terminal persistence\nbrowser-automation — Browser automation concepts\ntool-calling — Tool calling patterns\n\nDecisions\n\nindex — Decision index\nkeep-openrouter-as-fallback — Keep OpenRouter as fallback provider\npersistent-shell-enabled — Enable shell persistence\n\nComparisons\n\ncamoufox-vs-chrome — Camoufox vs Chrome\nhermes-vs-nanobot — Hermes vs Nanobot\nmemory-backends — Memory backend comparison\nmodel-routers — Model router comparison\n\nRunbooks\n\nmultiplexer-setup — Multi-profile gateway multiplexer setup\ndiagnose-docker — Docker diagnosis\nrecover-docker-service — Docker service recovery\nrestart-hermes — Hermes restart procedure\nrestart-browser — Browser restart procedure\nupdate-hermes-safely — Safe Hermes update\nrestore-openrouter — OpenRouter restoration\nprovider-health — Provider health check\ntroubleshoot-tool-loop — Tool loop troubleshooting\ngateway-resource-resilience — Gateway resilience\njob-radar — Job Radar search, Notion, dashboard and recovery procedure\n\nRaw\n\nhermes-config-sanitized.txt — Sanitized config\n"},"infrastructure/backups":{"slug":"infrastructure/backups","filePath":"infrastructure/backups.md","title":"Backups","links":["runbooks/update-hermes-safely","runbooks/backup-wiki","systems/mnemosyne","systems/hermes-agent"],"tags":["infrastructure","backups"],"content":"Backups\nPurpose\nHow Hermes’s own configuration/state gets backed up, and what’s missing.\nCurrent mechanism\n\nhermes backup CLI (verified via hermes backup --help): creates a zip of the entire Hermes config, skills, sessions, and data (excludes the hermes-agent codebase itself).\n\n--quick / -q: fast snapshot of just critical state (config, state.db, .env, auth, cron)\n--label / -l: label for a quick snapshot\n--output / -o: custom output path (default ~/hermes-backup-<timestamp>.zip)\n\n\nConfig keys (updates: section of ~/.hermes/config.yaml):\n\npre_update_backup: false — Hermes does not automatically back up before self-updates\nbackup_keep: 5 — retention count (applies to config.yaml.bak* rotation seen during updates, e.g. in update-hermes-safely)\nnon_interactive_local_changes: stash\n\n\n\nEvidence on disk (2026-07-22)\n\nOne manual full backup exists: /home/hermes/hermes-backup-2026-07-22-180830.zip (~235 MB), same day as wiki creation.\n/home/hermes/wiki-personal-backups/ holds an earlier snapshot of the wiki itself (SCHEMA.md, current-state.md, index.md, log.md from 17:34, before later same-day wiki edits) — see backup-wiki.\n/home/hermes/hermes-wiki/ exists but is empty — likely a staging/rename artifact from wiki setup, not an active backup target.\nA dedicated memory-backup script exists: /home/hermes/.hermes/scripts/mnemosyne-backup.sh — see mnemosyne.\n\nKnown gaps\n\nNo scheduled backup job found: crontab -l -u hermes returns empty, and no systemd timer references hermes or backup. The one full backup on disk appears to have been triggered manually, not on a schedule.\npre_update_backup: false means an in-place hermes update will not auto-snapshot first — the manual step in update-hermes-safely step 1 is currently the only safety net.\n\nRelevant runbooks\n\nbackup-wiki\nupdate-hermes-safely\n\nRelated\n\nmnemosyne\nhermes-agent\n"},"infrastructure/credentials-map":{"slug":"infrastructure/credentials-map","filePath":"infrastructure/credentials-map.md","title":"Credentials Map","links":["infrastructure/hosts","systems/model-providers","systems/messaging-integrations"],"tags":["credentials","security","infrastructure"],"content":"Credentials Map\nPurpose\nDocument where credentials are stored and what service they belong to.\nNo credential values are stored in this wiki.\nStorage Locations\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nServiceCredential StoreLocationNotesTelegram BotHermes config~/.hermes/config.yamlNot exposed in wikiDiscord BotHermes config~/.hermes/config.yamlNot exposed in wikiEmailHermes config~/.hermes/config.yaml-connected via IMAP/SMTPMattermostHermes config~/.hermes/config.yamlConnectedCF Access (mcp-portal)Hermes config~/.hermes/config.yamlClient ID + SecretLiteLLMHermes config~/.hermes/config.yamlAPI keyOmniRouteHermes config~/.hermes/config.yamlAPI keyAnthropicHermes config/OAuth~/.hermes/.anthropic_oauth.jsonAPI VaultHermes vaultvia hermes-api-vault skillNotion-backed vaultHuawei CloudTerraform state / configUnverifiedGitHubHermes config / gh CLI~/.config/gh/ or ~/.hermes/Unverified\nRules\n\nCredentials never written to this wiki\nAPI vault is first-stop for any key lookup\nIf vault lacks it, surface the gap to user — do not fabricate\n\nRelated\n\nhosts\nmodel-providers\nmessaging-integrations\n"},"infrastructure/docker-services":{"slug":"infrastructure/docker-services","filePath":"infrastructure/docker-services.md","title":"Docker Services","links":["systems/homepage-dashboard","infrastructure/docker-services","infrastructure/hosts"],"tags":["docker","homelab","infrastructure"],"content":"Docker Services\nPurpose\nContainerized services supporting Hermes and the homelab.\nStatus\n\nDocker runtime not available on this host (docker ps returned unavailable)\nCompose files confirmed present but execution state unverified\n\nConfirmed Compose Files\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFilePathServicesHermes Agent/home/hermes/.hermes/hermes-agent/docker-compose.ymlhermes, hermes-dashboardHermes WebUI/home/hermes/hermes-webui/docker-compose.ymlwebuiReadLater/home/hermes/workspace/readlater/docker-compose.ymlreadlater, edge-tts, kokoro-ttsSteel Browser/root/steel-browser/docker-compose.yml (CT201)steel-browser-api, steel-browser-ui\nService Definitions (from compose files)\nhermes-agent\n\nImages: hermes-agent\nContainers: hermes, hermes-dashboard\nVolumes: present (paths unverified)\nEnvironment: present (values unverified — secrets redacted)\n\nhermes-webui\n\nPorts: present (specific ports unverified)\nVolumes: present\n\nreadlater\n\nContainer: readlater\nPorts: present\nVolumes: present\nDependencies: edge-tts (travisvn/openai-edge-tts), kokoro-tts (ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.1)\n\nKnown Issues\n\nDocker runtime unavailable on inspection host — service state unknown\nSpecific ports, volumes, and env vars not extracted from compose files during this pass\n\nAdditional Services (CT201 runtipi)\n\nHomepage — Dashboard at port 8082 (Tailscale: runtipi.kangaroo-eel.ts.net:8443/)\nManager — Web UI for Homepage at port 8084\nSee systems/homepage-dashboard.md for full docs\n\nRelated\n\ndocker-services\nhosts\n"},"infrastructure/domains-and-tunnels":{"slug":"infrastructure/domains-and-tunnels","filePath":"infrastructure/domains-and-tunnels.md","title":"Domains and Tunnels","links":["systems/omniroute","comparisons/hermes-vs-nanobot","runbooks/diagnose-docker","systems/cloudflare-tunnel","infrastructure/network-map","infrastructure/hosts"],"tags":["infrastructure","network","cloudflare"],"content":"Domains and Tunnels\nPurpose\nWhere martinwa.org subdomains point, and how they reach internal services.\nHow it works\n\nVerified: All *.martinwa.org public hostnames are served by a single cloudflared daemon running on ct207 (webserver), tunnel name ct207-tunnel (ID 339a6757-5574-4723-a16c-da4769535caa), config at /etc/cloudflared/config.yml on ct207.\nVerified: cloudflared is not installed on this host (ct460/hermes) — Hermes has no local tunnel process to manage or restart. Any tunnel-side incident (hostname down, wrong target) must be fixed on ct207, not here.\nTwo other Cloudflare tunnels exist in the same account (Cloudo, Ifttt-ssh-webhook, livinity-liv) but are unrelated to Hermes/ct207 and out of scope for this page.\n\nHostnames relevant to Hermes\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nHostnameTargetNoteshermes.martinwa.orghttp://100.118.5.51:19119Public Hermes dashboard — nginx on ct460 proxies 19119→127.0.0.1:9119hermes-desktop.martinwa.orghttp://100.118.5.51:5174Hermes desktop companion; sets Host: hermes.kangaroo-eel.ts.nethermesagent.martinwa.orghttp://100.98.205.112:9119The older/original Hermes agent (ct450 “hermesagent”), kept alive alongside the current ct460 instancessh-hermes.martinwa.orgssh://100.118.5.51:22SSH to this host over the tunnelmcp-portal.martinwa.org(not in ct207 ingress list; see note below)Referenced in Hermes config as an MCP server URL (mcp-portal.martinwa.org/mcp)omniroute.martinwa.orghttp://omniroute.kangaroo-eel.ts.net:20128Public front for omniroutenanobot.martinwa.org / nano-ssh.martinwa.org100.122.189.81 (ct333 “clawtest”)See hermes-vs-nanobot\nmcp-portal.martinwa.org is not present in the ct207 ingress list captured during this audit — it may be served by a different tunnel/host, or the ingress list may have moved since. Unknown, flagged for follow-up rather than guessed.\nFull ct207 ingress list (context, not all Hermes-related)\nThe tunnel also fronts: pve.martinwa.org (Proxmox UI), docker.martinwa.org (Portainer), runtipi.martinwa.org, homar.martinwa.org, nextcloud.martinwa.org, readlater.martinwa.org, velvet.martinwa.org, overtchat.martinwa.org, freellmapi.martinwa.org, omnideb.martinwa.org, newstarter.martinwa.org, and SSH endpoints for debianvm/pve/webserver. These belong to the wider homelab, not Hermes specifically — recorded here only so an ingress diff is possible later.\nHistory\n\nct207’s /etc/cloudflared/ directory has ~25 dated config.yml.bak* files going back to 2026-06-18, including several during ct460’s own setup (config.yml.before-hermes-ct460-20260717203530, ...before-hermes-desktop-20260718, ...before-hermesagent-20260718085557) — evidence the ingress list has been actively iterated on throughout Hermes’s bring-up.\n\nRelevant runbooks\n\ndiagnose-docker\n\nRelated\n\ncloudflare-tunnel\nnetwork-map\nhosts\n"},"infrastructure/hosts":{"slug":"infrastructure/hosts","filePath":"infrastructure/hosts.md","title":"Known Hosts and Paths","links":["infrastructure/docker-services"],"tags":["infrastructure","hosts","containers","paths"],"content":"Known Hosts and Paths\nVerified Hosts\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nHostRoleAddressVerifiedlocalhost (this host)Primary Hermes runtime, Xvfb, file storage:102 displayYeslitellmLiteLLM proxyhttp://litellm:4000/v1Yes (from config)omniOmniRoute proxyhttp://omni:20128/v1Yes (from config, 2026-08-13)CT201Steel Browser, Runtipi, Homepage dashboard100.96.244.39 (Tailscale), 192.168.178.166 (LAN)Yes (live probe)CT460Hermes host, Camoufox, cua-driverhermes.kangaroo-eel.ts.netYesCT450 (hermesagent)Hermes Agent v0.19.0, hermes-webui, hermes-ui (Vite)100.98.205.112 (Tailscale), 192.168.178.65 (LAN)Yes\nProject Paths\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPathTypeNotes/home/hermesHermes home~/.hermes/ contains config, skills, cache/home/hermes/workspaceWorkspacedefault terminal cwd/home/hermes/.local/bin/hermesHermes binary/home/hermes/.hermes/config.yamlConfigversion 33/home/hermes/.hermes/skills/Skills dir66 skills installed/home/hermes/.hermes/cron/Cron jobs25 active jobs/home/hermes/.hermes/mnemosyne/Mnemosyne DBbackups/, data/, logs/, models//home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-drivercua-driverv0.9.0~/.camofox/Camofox profilesmanaged persistence~/.camoufox/Camoufox profilesmultiple profiles present/home/hermes/.hermes/browser_screenshots/Browser captures/home/hermes/wiki/WikiThis knowledge base/home/hermes/obsidian-vault/Obsidian vaultpersonal notes (50 - Research etc.)/home/hermes/hermes-webui/Hermes WebUIweb UI source/home/hermes/hermes-ui/Hermes Desktop browser (Vite dev)present on CT460 and CT450 port 5174/home/hermes/camofox-browser/Camofox browser/home/hermes/camoufox/Camoufox source/home/hermes/services/Services dir/home/hermes/www/Web root/home/hermes/go/Go workspace/home/hermes/Downloads/Downloads\nCompose Files (Docker runtime state unknown)\n\n/home/hermes/.hermes/hermes-agent/docker-compose.yml\n/home/hermes/hermes-webui/docker-compose.yml\n/home/hermes/workspace/readlater/docker-compose.yml\n\nHomelab References\n\nPVE host: pve.kangaroo-eel.ts.net (100.96.100.82) per session notes — Unverified on this host\nCT600: OmniRoute server (retired → CT601 omni, 2026-08-13) — Verified via pct list\n\nUnverified\n\nCT600, CT333, and other remote Proxmox containers/VMs referenced in session notes but not confirmed from this host\nPVE host identity not confirmed from this host without explicit confirmation\n\nRelated\n\ndocker-services\n"},"infrastructure/network-map":{"slug":"infrastructure/network-map","filePath":"infrastructure/network-map.md","title":"Network Map","links":["systems/cloudflare-tunnel","infrastructure/domains-and-tunnels","runbooks/update-hermes-safely","systems/headless-server","runbooks/diagnose-docker","runbooks/recover-docker-service","infrastructure/hosts"],"tags":["infrastructure","network"],"content":"Network Map\nPurpose\nWhere Hermes and its dependencies run, and how traffic reaches them.\nZones\nLAN\n\nCluster host (pve): 192.168.178.39 (Proxmox VE, kernel 6.17.4-1-pve)\nThis container (hermes, ct460): 192.168.178.129 (DHCP, eth0)\n\nTailscale (kangaroo-eel.ts.net)\n\npve: 100.96.100.82\nhermes (ct460): 100.118.5.51\nhermesagent (ct450, the original/legacy Hermes agent dashboard): 100.98.205.112\nomniroute (ct600): resolves via omniroute.kangaroo-eel.ts.net → 100.88.81.19\nnanobot (ct333, “clawtest”): 100.122.189.81\nwebserver (ct207, runs the Cloudflare Tunnel): 100.82.40.23\ndebianvm (VM 101): 100.98.95.65\ndocker host (ct200): 100.96.150.69\nopenweb (ct245): 100.69.230.13\nnextcloud (ct270): 100.90.102.19\n\nPublic (Cloudflare Tunnel)\n\nOrigin daemon runs on ct207 (webserver), not on this host. See cloudflare-tunnel and domains-and-tunnels for the full ingress list.\nThis host is reached publicly via two routes defined in that tunnel: hermes.martinwa.org → 100.118.5.51:19119 (nginx-fronted webui) and hermes-desktop.martinwa.org → 100.118.5.51:5174 (desktop companion).\n\nLocal ports on this host (ct460)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPortBindProcessPurpose22*sshdSSH80 / [::]:80*nginxreverse proxy, default vhost19119*nginx → 127.0.0.1:9119public Hermes webui path (tunneled)91190.0.0.0hermes (webui)Hermes Web UI backend8644 / 8642*hermesadditional Hermes agent ports5174*nodeHermes desktop companion (tunneled as hermes-desktop.martinwa.org)9377*nodeStratos — AI Command Centre8787127.0.0.1*pythonHermes gateway health endpoint (see update-hermes-safely)9099 / 8790*python3auxiliary Hermes processes (unverified purpose)5901 / [::]:5901*x11vncVNC on Xvfb :99, see headless-server6080 / 6081127.0.0.1websockifynoVNC web bridges for two VNC displays25127.0.0.1 / [::1]postfixlocal mail transport\n* curl target used in update-hermes-safely health check; binding not independently re-verified in this pass.\nRelevant runbooks\n\ndiagnose-docker\nrecover-docker-service\n\nRelated\n\nhosts\ndomains-and-tunnels\ncloudflare-tunnel\n"},"infrastructure/proxmox":{"slug":"infrastructure/proxmox","filePath":"infrastructure/proxmox.md","title":"Proxmox VE","links":["systems/homepage-dashboard","infrastructure/hosts","infrastructure/network-map"],"tags":["proxmox","virtualization","infrastructure","homelab"],"content":"Proxmox VE\nPurpose\nHypervisor hosting all VMs and LXC containers for the homelab.\nDetails\n\nHost: pve.kangaroo-eel.ts.net (Tailscale: 100.96.100.82)\nNode name: pve\nVersion: Proxmox VE 8.x\nPrimary user: Anthony@pam (full admin)\nAdditional users: root@pam, glance@pve, prom-exporter@pve, homepage@pve\n\nVMs (QEMU)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nVMIDNameStatusPurpose101debianvmrunningDebian VM — Portainer, filebrowser, code-server, Sunshine100homeassistantrunningHome Assistant282ubuntu-server-xfcerunningUbuntu desktop — Sunshine, Kubuntu403open-clawrunningOpenClaw agent401hermes-oldstoppedLegacy Hermes VM (deprecated)\nLXC Containers\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nVMIDNameStatusPurpose200dockerrunningMain Docker host — 42+ containers201runtipirunningRuntipi — Homepage, ntfy, etc.205ollamarunningOllama LLM inference + Guanaco proxy (v0.8.14, updated 2026-08-14)207webserverrunningWeb server — Vercel AI, manifest router215n8nrunningN8n automation221gitearunningGitea git server241ntfyrunningNtfy push notifications245openwebrunning255mattermostrunningMattermost chat260freshrssrunningFreshRSS feed reader270nextcloudrunningNextCloud299sftpgorunningSFTPGo file server — security review 2026-08-09: connects as root, unpinned fingerprint, // root path310monitoringstoppedMonitoring stack (deprecated)333clawtestrunningClawTest — Nanobot, Zeroclaw450hermesagentrunningHermes Agent LXC460hermesrunningHermes CT460465qwenpawrunningQwenPaw LLM500litellmrunningLiteLLM proxy555elementsynapsestoppedElement Synapse (deprecated)600omniroutestoppedOmniRoute model router — retired 2026-08-13, migrated to CT601601omnirunningOmniRoute model router — successor to CT600, since 2026-08-13103cliproxyapirunningCliproxy API104rclonerunningRclone sync106glancestoppedGlance dashboard209cloudrevestoppedCloudreve (deprecated)\nAPI Tokens\n\nhomepage@pve!homepage — PVEAuditor role, privsep=0 (used by Homepage dashboard)\n\nKey Paths\n\n/etc/pve/ — PVE config directory\n/var/lib/vz/ — Local storage\n/etc/pve/priv/token/ — API token storage\n\nRelated\n\nhomepage-dashboard — Dashboard shows PVE stats\nhosts — Full host inventory\nnetwork-map — Network topology\n"},"infrastructure/tailscale":{"slug":"infrastructure/tailscale","filePath":"infrastructure/tailscale.md","title":"Tailscale","links":["infrastructure/domains-and-tunnels","systems/homepage-dashboard","runbooks/job-radar"],"tags":["tailscale","vpn","networking","infrastructure"],"content":"Tailscale\nPurpose\nWireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access via Tailscale Serve.\nDomain\n\nTailnet: kangaroo-eel.ts.net\nPrimary node: CT460 (hermes.kangaroo-eel.ts.net)\nRuntipi node: CT201 (runtipi.kangaroo-eel.ts.net)\n\nCT460 Serve Routes\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPathTargetNotes/http://127.0.0.1:9119Hermes WebUI/jobshttp://127.0.0.1:9099Jobs Dashboard/desktophttp://127.0.0.1:6081CUA noVNC/vnc-camofoxhttp://127.0.0.1:6080Camofox VNC\nCT460 Funnel Routes\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPortTargetNotes:8443/webhookhttp://127.0.0.1:8085/webhookPublic webhook Funnel\nCT201 Serve Routes\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nPathTargetNotes:8443http://localhost:8082Homepage dashboard/steelhttp://127.0.0.1:5173Steel Browser UI/steel-apihttp://127.0.0.1:3000Steel Browser API (path prefix not stripped — use direct IP for API calls)\nAtomic Restore Script\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh — Rebuilds the complete CT460 route table, including tailnet Serve on :443 and public Funnel on :8443.\ntailscale serve reset clears Funnel state as well as Serve state. The atomic script therefore restores /webhook after rebuilding the 443 routes. Do not use the older Serve-only restore path for complete recovery.\nKey Commands\n# Check routes\ntailscale serve status\n \n# Rebuild routes (CT460)\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh\n \n# Add route\ntailscale serve --set-path /jobs http://127.0.0.1:9099\n \n# Remove route\ntailscale serve --set-path /jobs off\nNotes\n\nPort 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there\nCT460 has no conflict — Tailscale uses 443 directly\nRoutes must be re-added in order (restore script handles this)\nAfter every rebuild, verify that tailscale serve status contains both the :443 Serve routes and the :8443/webhook Funnel route.\n\nRelated\n\ndomains-and-tunnels — Full DNS and tunnel inventory\nhomepage-dashboard — Homepage on CT201 via Tailscale\njob-radar — Dashboard incident and route verification\n"},"log":{"slug":"log","filePath":"log.md","title":"log","links":["runbooks/job-radar","incidents/2026-08-01-mnemosyne-capture-failure","systems/mnemosyne","systems/memory-backend","comparisons/steel-vs-camoufox","systems/browser-backend","current-state","concepts/browser-automation","decisions/rollback-steel-to-camofox","systems/scheduled-tasks","systems/model-providers","reports/uber-eats/","decisions/","/","runbooks/multiplexer-setup","incidents/2026-08-04-omniroute-free-deepseek-silent-stops","systems/omniroute","systems/nanobot","infrastructure/proxmox","comparisons/memory-backends","comparisons/model-routers","infrastructure/hosts","systems/guanaco"],"tags":[],"content":"Wiki Log\n\nChronological record of wiki actions. Append-only.\n\n[2026-07-22] init | Fresh Hermes Systems Wiki\n\nCreated technical wiki at /home/hermes/wiki\nStructure: systems/, infrastructure/, concepts/, decisions/, comparisons/, runbooks/, experiments/, queries/, raw/, _archive/\nWrote SCHEMA.md with 11 operating rules\nWrote index.md as curated Start here dashboard\nReseeded system/infrastructure/runbook stubs for existing homelab\nGit initialized, local-only\nNo secrets, passwords, tokens, or private keys written to vault\n\n[2026-07-22] audit | External audit pass (Claude, at Anthony’s request)\n\nMerged duplicate pages: systems/omniRoute.md (empty stub) -> kept systems/omniroute.md (populated); repointed 5 links\nMerged duplicate pages: runbooks/safely-update-hermes.md -> kept as runbooks/update-hermes-safely.md (fuller content); repointed 2 links\nAdded missing YAML frontmatter to 19 pages that lacked it (SCHEMA rule violation)\nFlagged 17 skeleton/placeholder pages explicitly as stubs (status: unresolved, confidence: low) instead of leaving them silently thin\nLinked 5 previously-orphaned pages (network-map, domains-and-tunnels, backups, cloudflare-tunnel, home-assistant) into index.md so they’re reachable from Start Here\nVerified: no secrets/keys/passwords found in any wiki page (rule 1 compliance holds)\nDid not fabricate comparison/concept content — left for Hermes/Anthony to fill with verified facts\n\n[2026-07-22] populate | Fill stub pages with verified system facts (Claude, at Anthony’s request)\n\nPopulated with real, verified data (not fabricated): infrastructure/network-map, infrastructure/domains-and-tunnels, infrastructure/backups, systems/cloudflare-tunnel, systems/home-assistant, systems/headless-server, systems/hindsight, comparisons/camoufox-vs-chrome, comparisons/model-routers, comparisons/hermes-vs-nanobot, comparisons/memory-backends, concepts/persistent-shells, concepts/tool-calling, concepts/browser-automation, concepts/agent-memory\nCorrected systems/nanobot.md: Nanobot is actually running, on LXC 333 (“clawtest”), not this host — earlier “not running” claim was true only for ct460 and got conflated with pve’s own IP\nFound and flagged a real inconsistency: decisions/use-hindsight-for-memory.md still names Hindsight as primary backend while current-state.md/systems/memory-backend.md say Mnemosyne — marked unresolved rather than silently overwritten\nFixed a broken review_after: 0001-01-01 in decisions/keep-openrouter-as-fallback.md\nNew security-relevant findings recorded in-place: x11vnc on this host has no password and binds to all interfaces (systems/headless-server.md)\nqueries/retained-research-answers.md intentionally left as a stub — it’s a running research log to be filled by Hermes over time, not something system inspection can populate\n\n[2026-07-22] correction | Purge Hindsight (per Anthony, direct correction)\n\nAnthony confirmed Hindsight has nothing to do with this Hermes box (ct460) — it was confused with a different Hermes instance during earlier wiki authoring\nDeleted: systems/hindsight.md, decisions/use-hindsight-for-memory.md, runbooks/check-hindsight.md\nRewrote comparisons/memory-backends.md and concepts/agent-memory.md to drop Hindsight as this box’s predecessor backend\nRemoved Hindsight references from: systems/memory-backend.md, systems/hermes-agent.md, systems/mnemosyne.md, current-state.md, decisions/index.md, index.md\nLeft in place (not deleted, just flagged): archived hindsight-*.py scripts under ~/.hermes/scripts/ on this host — their presence is noted in comparisons/memory-backends.md as unexplained, possibly generic install-template leftovers, not evidence of local use\nConfirmed: Mnemosyne is, and has only ever been, this box’s memory backend\n\n[2026-07-22] cleanup | Remove leftover Hindsight scripts (per Anthony)\n\nDeleted /home/hermes/.hermes/scripts/hindsight-watchdog.py.archived, hindsight_migration_watchdog.py.archived, honcho_to_hindsight_import.py.archived\nThese were the last physical trace of the earlier Hindsight/another-Hermes mix-up (see prior wiki + decisions correction); nothing Hindsight-related remains on this host or in this wiki\n\n[2026-07-25] operations | Tool Search and gateway resilience\n\nDocumented always-on native Hermes Tool Search, its verified context reduction, new-tool behavior, and rollback backup\nDocumented how new skill categories are derived from the first folder below ~/.hermes/skills/\nDocumented CT460 gateway reclaim/CPU protection and the memory containment applied to Camofox and Hermes Web UI\nRecorded the I/O-controller delegation limitation so an ineffective IOWeight is not mistaken for active protection\nAdded health checks, exact effective settings, backup location, and rollback procedure\nNo secrets, tokens, or private credentials were written to the wiki\n\n[2026-07-28] dashboard | Jobs Dashboard feature additions\n\nFixed bulk status update (payload shape mismatch: {ids, status} → {ids, updates: {status}})\nFixed column header sorting (sort key suffix mismatch, added all 8 column sort handlers)\nAdded Phone Screen as a dedicated KPI filter card (separated from Interviews)\nAdded Market Intel section at top of page (lazy-loaded on expand, refreshes on Refresh)\nAdded rich text notes editor (Write/Preview with markdown: bold, italic, links, bullets) syncing to Notion Notes / JD Summary\nAdded document attachment system (bookmark blocks on Notion page body, lazy-loaded, add/delete via API)\nAdded 2 strong-match roles manually: Steel Blue Marketing Manager – APAC, Revo Fitness Campaign Manager\nUpdated Synergy and Dept of Communities to Phone Screen status\nUpdated wiki runbook with full dashboard feature documentation and API endpoint table\n\n[2026-07-27] incident | Job Radar search and dashboard recovery\n\nDocumented the contaminated morning radar run, 16-record cleanup, deterministic seven-day freshness/direct-advert rules, Pacific Energy exclusion, full Notion pagination and strong-match cover-letter workflow\nAdded the non-negotiable same-day cover-letter self-healing audit: retry once, verify all five artifacts, require canonical HTML/HTTP 200, and require MISSING_COVER_LETTER_COUNT=0\nRecorded the production cron model (gpt-5.6-terra) and verified first inherited fallback (opencode-go/mimo-v2.5)\nDocumented dashboard direct-advert links, one-click KPI/stat/funnel filters, and retained Notion-backed status editing\nRecorded the CT460 Tailscale route-table loss and corrected atomic restoration of both :443 Serve and :8443/webhook Funnel\nAdded job-radar and updated scheduled-task, Tailscale and index references\nVerified no secrets, API keys or private credentials were written to the wiki\n\n[2026-07-29] update | Steel Browser deployment — browser backend switch\n\nNew page: systems/browser-backend.md — rewritten to reflect Steel Browser as primary, Camoufox as fallback\nNew page: comparisons/steel-vs-camoufox.md — comparison of the two backends\nUpdated: infrastructure/hosts.md — added CT201 and CT460 to verified hosts table\nUpdated: infrastructure/docker-services.md — added Steel Browser compose file\nUpdated: infrastructure/tailscale.md — added CT201 Steel serve routes\nUpdated: index.md — browser-backend link description\nUpdated: current-state.md — added 2026-07-29 changes section\nConfig change: browser.cdp_url set to http://100.96.244.39:9223 in ops profile\n\n[2026-07-30] update | Open WebUI updated on CT245\n\nUpdated image to ghcr.io/open-webui/open-webui:main (new digest e97bf9531916, Jul 27)\nBumped CT245 memory from 2592→4096 MB, swap 1600→2048 MB to prevent OOM on latest image\nRestarted container after OOM hang\nUpdated infrastructure/proxmox.md\n\n[2026-08-02] incident | Mnemosyne embedding coverage gap discovered and fixed\n\nHealth monitor was reporting vector index counts but not actual embedding coverage, hiding a critical gap\nWorking memory embeddings were at 12.4% (186/1,504) — early memories from July 17-26 were never embedded\nRan mnemosyne reindex --yes to rebuild all embeddings — working memory now at 100%\nUpdated health check script (mnemosyne-health-check.sh) to report actual embedding coverage via new helper script (check_embedding_coverage.py)\nCreated embedding coverage monitoring cron job (b63ed5817212) — runs every 6 hours, alerts if working memory coverage drops below 95%\nAdded auto-reindex rule to MEMORY.md — will run reindex automatically if coverage drops, no user prompt needed\nDB grew from 8.7M to 21M with the new embeddings\n\n[2026-08-01] incident | Mnemosyne capture pipeline repaired\n\nConfirmed automatic conversation capture failure with a pre-fix unique-marker E2E test: Hermes completed the turn but no working_memory row existed.\nCorrected stringified structured YAML (sync_roles, skip_contexts, reflect, sleep_threshold, ignore_patterns) and enabled profile isolation; hermes config check passed.\nVerified the repair with a second CLI marker that persisted as an exact normalized [USER] database row.\nCorrected stale canonical model facts for light-only design, numbered Obsidian structure, and CT460 root route :9119; added authoritative identity/preference/workflow slots.\nReduced legacy always-injected memory files from 8,013 bytes to 1,080 bytes, with timestamped backups retained.\nAdded 2026-08-01-mnemosyne-capture-failure and refreshed mnemosyne plus memory-backend.\n\n[2026-08-02] update | Wiki audit and Steel→Camoufox rollback documentation\n\nRewrote steel-vs-camoufox — full Steel experiment story, timeline, lessons learned\nRewrote browser-backend — Camoufox primary, Steel disabled\nUpdated current-state — browser revert, Mnemosyne vector gap, DeepSeek 0731, Uber Eats upgrade, GH Trending removal, Tembo→Tusk rename\nUpdated browser-automation — added Steel experiment lesson\nCreated rollback-steel-to-camofox — formal decision record\nUpdated scheduled-tasks — job count 25, Uber Eats upgrade, GH Trending Daily removed\nUpdated mnemosyne — vector embedding coverage gap documented\nUpdated model-providers — DeepSeek V4 Flash 0731 details\nCreated index — landing page for monthly spend reports\nUpdated index — linked new decision page\nUpdated index.md — browser description, reports section, last-updated date\nCreated nightly wiki audit cron job (23:59 AWST)\n\n[2026-08-02] update | Ops profile Mnemosyne enabled and wiki sync\n\nOps profile (~/.hermes/profiles/ops/config.yaml): enabled mnemosyne plugin with profile_isolation: true, allow_tool_override: false, shared_surface_read: false\nTusk (ops) and Rhino (default) now have independent memory banks — zero cross-contamination\nVerified end-to-end: memory writes land correctly in both profiles independently\nUpdated mnemosyne — profile isolation section now documents ops config\nUpdated current-state — added ops profile Mnemosyne entry\nUpdated multiplexer-setup — fixed “Elephant ops” → “Tusk ops” in architecture diagram\nFixed memory-backend — “Shared surface empty” section renamed to reflect intentional isolation decision\n\n[2026-08-04] update | OmniRoute free-DeepSeek silent-stop mitigation\n\nIncident: free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances\nReworked deepseek-flash-free and good-free on CT600: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort\nCreated 2026-08-04-omniroute-free-deepseek-silent-stops\nUpdated current-state — added OmniRoute free-DeepSeek mitigation to Recent Changes\nUpdated omniroute — added Recent Changes (2026-08-04) section\nUpdated model-providers — added mitigation entry under Notable Model Updates, updated updated date\n\n[2026-08-07] update | Nightly wiki audit — OmniRoute fix, Nanobot degraded, scheduled tasks rewrite\n\nOmniRoute three-layer fix documented: npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall pattern. is_active=0 does NOT stop CredentialHealth probes. All lessons banked in omniroute-ops skill.\nUpdated omniroute — added Recent Changes (2026-08-07) section with full root cause analysis\nUpdated nanobot — status changed from “active” to “degraded” (DOWN, connection refused on 8900). Updated confidence, sources, verified_on.\nUpdated current-state — added 2026-08-07 changes: OmniRoute fix, Nanobot degraded, CT460 memory bump\nRewrote scheduled-tasks — fixed duplicate entries (5 duplicates removed), added 8 missing jobs (Important-mail monitor, CT221 Miner Watchdog, Job Tracker Dashboard Refresh, Evening Stand-Down Briefing, Daily Birthday Checker, Mnemosyne Embedding Coverage Monitor, Hermes Config Backup, Review Agent Estate). Count corrected: 25 → 28.\nWiki audit cron error noted: model drift (smart-route → deepseek-paid-smart) causing RuntimeError on Wiki Audit job\nUpdated index.md — last-updated date 2026-08-04 → 2026-08-07\n\n[2026-08-09] update | Nightly wiki audit — Portainer reset, SFTPGo review\n\nPortainer password reset documented: CT245 admin user confirmed as anthony (not admin). Password reset via helper container, verified with auth API. Recovery procedure recorded. Added to current-state.\nSFTPGo security review documented: CT299 findings — root access, unpinned fingerprint, // root path, F1 quota mystery. Added to current-state and proxmox.\n\n[2026-08-08] update | Cron model repinning, hermes-ui revert, Phase One review\n\nCron job model repinning: 13 jobs repinned from dead/stale provider combos. 3 from freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker from freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named jobs (e.g. vertex/gemini-2.5-flash, deepseek-v4-flash-free) → smart-route. Daily Marketing Job Radar → gpt-5.6-terra.\nHermes UI vite config revert: Vite config was changed to HTTPS with self-signed certs (~16:00), breaking iPad access. Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely CT450), not fixable from CT460.\nPhase One estate review completed: Manual cron run (ID a9359b7db27b) verified CT450’s 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460’s 11 production jobs remain enabled with last_status=ok, CT333 Nanobot has live jobs.json with 9 jobs. Cron review report contained two inaccuracies (wrong date, false claim about CT333) — verified via live system probes.\nUpdated current-state — added 2026-08-08 changes section\nUpdated scheduled-tasks — Phase One review marked completed, model repinning noted in Known Issues\n\n[2026-08-10] audit | Nightly wiki audit — cron failures, session storage issue\n\nCron job failures documented: Four jobs failed on 2026-08-10 with two error patterns:\n\nTimeoutError (idle 601–602s): Daily Marketing Job Radar, Political News Digest\nRuntimeError (session storage write failure): Tech & AI Newsletter Digest, GitHub Trending Radar — Weekly Roundup\n\n\nRoot cause identified: Disk at 83% (11GB free on 65GB), state.db at 593MB. Session storage likely experiencing WAL checkpoint or disk pressure issues.\nUpdated current-state — added 2026-08-10 changes section with cron failures and OpenCode WebSocket errors\nUpdated scheduled-tasks — added session storage failures to Known Issues, updated verified_on to 2026-08-10\nOpenCode WebSocket proxy errors noted: Repeated “OpenCode service unavailable” in hermes node journal (Aug 11 00:08). Process running, likely transient.\nNo new pages created — all changes applied to existing pages\n\n[2026-08-12] update | Nightly wiki audit — OmniRoute AutoRoutes, OpenViking evaluation, FreeLLMAPI internals\n\nOmniRoute AutoRoute variants documented: auto/best-free (Auto Cheap) and auto/coding-free are virtual variants — not among the 54 persisted combos. LKGP engine resolves them dynamically by scoring live connected accounts (antigravity, opencode-zen, nvidia, openrouter) on health/quota/latency/errors/cost-inverse, with instant failover. Example observed: auto/best-free routed to Antigravity. Updated omniroute.\nOpenViking memory provider evaluated, NOT adopted: single-slot config would fully replace Mnemosyne (no parallel mode); risk assessment flagged schema/embedding/scope/provenance loss and no rollback. Config remains memory.provider: mnemosyne. Added to memory-backends.\nFreeLLMAPI router internals documented: 3-axis weighted scoring (Reliability/Speed/Intelligence) + guardrail multiplier + Penalty Inspector + vision/tool capability matching. Router behind Hermes’s default freellmapi-auto. Added to model-routers.\nNo new pages created — all changes applied to existing pages. Routine cron (Mnemosyne health, embedding coverage, CT221 miner watchdog, important-mail monitor) passed clean; no incidents.\n\n[2026-08-13] update | Nightly wiki audit — OmniRoute CT600→CT601 migration\n\nOmniRoute migrated from CT600 to CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. All facts verified live (pct list, grep, Tailscale DNS).\nUpdated omniroute — Current Configuration rewritten for omni/CT601, added Recent Changes (2026-08-13), bumped updated/verified_on to 2026-08-13.\nUpdated proxmox — CT600 marked stopped/retired, CT601 added running; bumped updated.\nUpdated current-state — attested provider/host facts updated, added Recent Changes (2026-08-13) including evening stand-down weather bugfix.\nUpdated hosts — OmniRoute proxy entry → omni; CT600 reference marked verified/retired.\nUpdated index — last-updated date 2026-08-10 → 2026-08-13.\nNo new pages created; all changes applied to existing pages. No secrets written.\n\n[2026-08-14] update | Nightly wiki audit — OmniRoute factory reset, Guanaco update, GPT Luna research\n\nOmniRoute factory reset documented: CT601 database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at /opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys invalid — removed. Multiple free combos broken by upstream model ID changes (OpenRouter Poolside dash→dot, Groq llama-4-scout retired). Default model changed from auto/best-free to resilient-free. Working free combos: free-lkgp, mega-free, smart-route. See omniroute.\nGuanaco router documented: CT205 ollama container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. Created guanaco.\nGPT-5.6 Luna research documented: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna for planning/review, DeepSeek for execution. Updated model-providers.\nUpdated omniroute — added Recent Changes (2026-08-14), bumped updated/verified_on\nUpdated model-providers — provider table (omniroute → omni), Guanaco entry, GPT-5.6 Luna entry, default model changed to resilient-free, bumped dates\nCreated guanaco — new page for Guanaco LLM proxy on CT205\nUpdated current-state — added Recent Changes (2026-08-14) with all seven items, bumped updated\nUpdated proxmox — CT205 description updated with Guanaco proxy\nUpdated hosts — bumped updated date\nUpdated index — last-updated date → 2026-08-14, added Guanaco link to Core systems\nNo secrets written.\n"},"queries/retained-research-answers":{"slug":"queries/retained-research-answers","filePath":"queries/retained-research-answers.md","title":"Retained Research Answers","links":[],"tags":["query","research"],"content":"\nStub — structure only. Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.\n\nRetained Research Answers\nFiled research answers worth reusing."},"reports/uber-eats/2026-07":{"slug":"reports/uber-eats/2026-07","filePath":"reports/uber-eats/2026-07.md","title":"2026-07","links":[],"tags":[],"content":"🍔 Uber Eats Monthly Spend — 2026-07\nGenerated: 2026-08-01T03:30:00+08:00\nSource: Camofox authenticated Uber Eats session (profile anthony) via getPastOrdersV1 API\nSummary\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nMetricValueTotal Spend$264.51 AUDTotal Orders11Avg per Order$24.05Most Expensive$40.73Least Expensive$0.00\nvs Previous Month (2026-06)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nThis MonthLast MonthChangeSpend$264.51$118.05+$146.46Orders113+8Avg/Order$24.05$39.35$-15.30\nTop 5 Most Expensive Orders\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n#DateRestaurantAmount12026-07-11 17:02Coles$40.7322026-07-09 15:42Foodies Market IGA$36.3532026-07-23 20:31IGA Grocery$34.3842026-07-06 14:19806 Noodle House$24.6652026-07-09 12:56Lepak Kopitiam Vic Park, also known as Yum Yum Tree Vic Park$24.28\nMost Frequented Restaurants\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nRestaurantOrdersHungry Jacks3Angelo Street Market2806 Noodle House2IGA Grocery1Coles1Foodies Market IGA1Lepak Kopitiam Vic Park1\nAll Orders\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nDateTimeRestaurantAmount2026-07-0319:16Hungry Jacks$21.112026-07-0614:19806 Noodle House$24.662026-07-0814:00Angelo Street Market$24.072026-07-0912:56Lepak Kopitiam Vic Park, also known as Yum Yum Tree Vic Park$24.282026-07-0915:42Foodies Market IGA$36.352026-07-1117:02Coles$40.732026-07-1414:35Hungry Jacks$11.202026-07-1615:28806 Noodle House$23.662026-07-2014:58Angelo Street Market$24.072026-07-2320:31IGA Grocery$34.382026-07-2512:58Hungry Jacks$0.00"},"reports/uber-eats/index":{"slug":"reports/uber-eats/index","filePath":"reports/uber-eats/index.md","title":"Uber Eats Monthly Reports","links":["reports/uber-eats/2026-07","systems/scheduled-tasks"],"tags":["reports","uber-eats","finance"],"content":"Uber Eats Monthly Reports\nMonthly spend reports generated from Camofox-authenticated Uber Eats sessions.\nHow it works\n\nCron job runs on the 1st of each month at 02:00 AWST\nOpens a Camofox tab (profile anthony) and navigates to ubereats.com/orders\nCaptures the getPastOrdersV1 API response\nSaves JSON to ~/.hermes/uber_eats_report.json\nGenerates HTML report + wiki markdown page\nDelivers a formatted Telegram digest\n\nReports\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nMonthSpendOrdersAvg/OrderReport2026-07$264.5111$24.052026-07\nRelated\n\nscheduled-tasks\n"},"runbooks/backup-wiki":{"slug":"runbooks/backup-wiki","filePath":"runbooks/backup-wiki.md","title":"Backup Wiki","links":["runbooks/update-hermes-safely","current-state"],"tags":["runbook","wiki","backup"],"content":"Backup Wiki\nPurpose\nCreate a durable backup of the wiki so recoverable copies exist before mutating changes.\nSymptoms that match this runbook\n\nBefore a big restructuring, page rewrite, or history migration\nWeekly or scheduled backup reminder\nSuspected accidental deletion or corruption\n\nPrerequisites\n\nWiki path: /home/hermes/wiki\nGit is available\nBackup destination is writable\n\nProcedure\n1. Sanity-check repo state\ncd /home/hermes/wiki && git status --short && git log --oneline -3\n2. Commit any open changes first\ncd /home/hermes/wiki\ngit add -A\ngit commit -m "backup-baseline: $(date +%Y-%m-%d)"\n3. Export a timestamped copy outside the repo\nBACKUP_DIR=/home/hermes/wiki-backups\nmkdir -p "$BACKUP_DIR"\nBACKUP="$BACKUP_DIR/wiki-$(date +%Y%m%d-%H%M%S).tar.gz"\ntar -czf "$BACKUP" -C /home/hermes wiki\necho "Wrote $BACKUP"\n4. Optional: push to a Git remote if configured\ncd /home/hermes/wiki\ngit remote -v\ngit push --all\ngit push --tags\nIf no remote is configured, skip this step and rely on the tarball.\n5. Rotate old backups if needed\nKeep N most recent backups:\nls -1t "$BACKUP_DIR"/wiki-*.tar.gz | tail -n +6 | xargs -r rm\nVerification\n\ngit status --short shows a clean tree\nTarball exists and is non-empty:\n\nstat "$BACKUP"\ntar -tzf "$BACKUP" | head -20\nRollback\n\nRestore from tarball:\n\nBACKUP=$(ls -1t /home/hermes/wiki-backups/wiki-*.tar.gz | head -1)\nrm -rf /home/hermes/wiki-restore\nmkdir -p /home/hermes/wiki-restore\ntar -xzf "$BACKUP" -C /home/hermes wiki-restore --strip-components=1\n\nVerify restored content before replacing live wiki\n\nNotes\n\nThis runbook intentionally does not write secrets or token dumps into backups\nIf you need a full Hermes config+data backup, use hermes backup in addition to this wiki-only backup\n\nLast tested\n2026-07-22\nRelated\n\nupdate-hermes-safely\ncurrent-state\n"},"runbooks/diagnose-docker":{"slug":"runbooks/diagnose-docker","filePath":"runbooks/diagnose-docker.md","title":"Diagnose Failed Docker Service","links":["infrastructure/docker-services","infrastructure/hosts"],"tags":["runbook","docker","homelab"],"content":"Diagnose Failed Docker Service\nPurpose\nDiagnose container or compose issues on hosts where Docker is present.\nSymptoms that match this runbook\n\nService reports container exit/failed state\ndocker ps does not show an expected container\nCompose stack reports unhealthy status\n\nPrerequisites\n\ndocker installed and accessible to the user\nDocker daemon running\nCompose file path known for the affected project\n\nProcedure\n1. Check Docker daemon state\nsystemctl status docker\n2. List containers\ndocker ps -a --format "table {{.Names}}\\t{{.Image}}\\t{{.Status}}\\t{{.Ports}}"\n3. Inspect logs for the failing container\ndocker logs --tail 100 <container-name>\n4. If using compose, check compose state\ncd /home/hermes/<project>/docker-compose\ndocker compose ps\ndocker compose logs --tail 100 <service-name>\n5. Restart the failed container only\ndocker restart <container-name>\n6. If restart fails, investigate restart policy and image\ndocker inspect --format "{{.HostConfig.RestartPolicy}}" <container-name>\ndocker inspect --format "{{.Config.Image}}" <container-name>\nVerification\n\ndocker ps shows the container Up or healthy\nService or WebUI dependent on the container is reachable again\nLogs show clean startup messages\n\nRollback\n\nDo not delete volumes unless confirmed not in use\nIf a container recreate is required, preserve volume mappings and env files\nIf uncertain, collect logs first and ask before replacement\n\nNotes\n\nThis runbook is generic. This current Hermes host does not have Docker installed as of 2026-07-22\nDocker-related paths must be confirmed before executing compose commands\nDo not invent container names or project paths\n\nLast tested\n2026-07-22 on system where Docker runtime is absent; read-only steps verified, restart steps not executed\nRelated\n\ndocker-services\nhosts\n"},"runbooks/gateway-resource-resilience":{"slug":"runbooks/gateway-resource-resilience","filePath":"runbooks/gateway-resource-resilience.md","title":"Hermes Gateway Resource Resilience","links":["systems/hermes-agent","systems/messaging-integrations","systems/browser-backend","runbooks/restart-hermes","systems/tool-search"],"tags":["hermes","infrastructure","runbook"],"content":"Hermes Gateway Resource Resilience\nPurpose\nKeep Telegram and the Hermes gateway available when the browser or Web UI\ncauses severe memory pressure inside CT460.\nThe observed failure was real: the CT460 memory cgroup OOM-killed a large Web\nUI process and a Chrome child. Because the gateway previously used\nOOMPolicy=stop, the Chrome child death caused systemd to stop the entire\ngateway until Restart=always brought it back.\nActive controls\nGateway\nFile:\n/home/hermes/.config/systemd/user/hermes-gateway.service.d/resource-resilience.conf\n[Service]\nOOMPolicy=continue\nMemoryLow=512M\nCPUWeight=1000\nThe gateway has no MemoryHigh or MemoryMax. It receives reclaim protection\nand maximum relative CPU weight, while OOMPolicy=continue prevents an\nOOM-killed browser child from stopping the whole unit. If the main gateway\nprocess exits, its existing Restart=always policy still restarts it.\nCamofox\nFile:\n/home/hermes/.config/systemd/user/camofox-browser.service.d/resource-containment.conf\n[Service]\nMemoryHigh=1G\nMemoryMax=1536M\nMemorySwapMax=768M\nOOMPolicy=kill\nHermes Web UI\nFile:\n/etc/systemd/system/hermes-webui.service.d/resource-containment.conf\n[Service]\nMemoryHigh=1G\nMemoryMax=1536M\nMemorySwapMax=1G\nOOMPolicy=kill\nCamofox and Web UI are allowed to restart under extreme growth instead of\nconsuming most of the container and taking Telegram down with them.\nI/O priority limitation\nDo not add IOWeight to the gateway drop-in unless Proxmox cgroup delegation\nis changed and verified. CT460 exposes the I/O controller at its top cgroup,\nbut it is not delegated into the container’s systemd service tree. Systemd\naccepts the property, but no live io.weight control is created.\nHealth check\nsystemctl is-active hermes-webui.service\nrunuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \\\n  systemctl --user is-active camofox-browser.service hermes-gateway.service\nCheck effective limits:\nsystemctl show hermes-webui.service \\\n  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax\nrunuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \\\n  systemctl --user show hermes-gateway.service \\\n  -p OOMPolicy -p MemoryLow -p MemoryHigh -p MemoryMax -p CPUWeight\nrunuser -u hermes -- env XDG_RUNTIME_DIR=/run/user/1000 \\\n  systemctl --user show camofox-browser.service \\\n  -p OOMPolicy -p MemoryHigh -p MemoryMax -p MemorySwapMax\nExpected gateway values include OOMPolicy=continue,\nMemoryLow=536870912, MemoryHigh=infinity, MemoryMax=infinity, and\nCPUWeight=1000.\nBackups and rollback\nOriginal source units are backed up at:\n/home/hermes/.hermes/backups/resource-hardening-20260725/\nTo roll back, remove only these three drop-ins:\n\nresource-resilience.conf for hermes-gateway.service\nresource-containment.conf for camofox-browser.service\nresource-containment.conf for hermes-webui.service\n\nThen reload the system and user systemd managers and restart only those three\nservices. The original source units were never modified.\nRelated\n\nhermes-agent\nmessaging-integrations\nbrowser-backend\nrestart-hermes\ntool-search\n"},"runbooks/job-radar":{"slug":"runbooks/job-radar","filePath":"runbooks/job-radar.md","title":"Daily Marketing Job Radar Operations and Recovery","links":["systems/scheduled-tasks","infrastructure/tailscale","systems/model-providers"],"tags":["automation","cron","jobs","notion","dashboard","runbook"],"content":"Daily Marketing Job Radar Operations and Recovery\nPurpose\nRun a verified Western Australian marketing-job search, deduplicate against the complete Notion board, create only fresh direct adverts, score new roles against Anthony’s resume, automatically produce cover letters for strong matches, and expose the results through the Jobs Radar dashboard.\nProduction configuration\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSettingValueCron jobdca8482e4f76 — Daily Marketing Job RadarSchedule0 7 * * 1-5 (07:00 AWST weekdays)DeliveryTelegram group -1004321904721, topic 1030Skilljob-search-automationPrimary modelopenai-codex / gpt-5.6-terraFirst fallbackopencode-go / mimo-v2.5Dashboardhermes.kangaroo-eel.ts.net/jobs/Dashboard backend127.0.0.1:9099Notion database0f90ba2b-8b10-4d02-a62c-6f692d5b1168\nCron jobs inherit the profile-level fallback_providers chain; Hermes does not currently store a separate fallback chain in each cron record. MiMo v2.5 is first in the global chain, so Terra fails over to it before the older fallbacks.\nCover letters are non-negotiable\nA successful radar run includes the cover-letter stage.\n\nAudit every Strong Match added today, including pages created by an interrupted earlier run.\nA valid result requires a canonical rendered .html URL in Notion, not plain text and not the dashboard root.\nProduction template files:\n\n/home/hermes/.hermes/dashboard/templates/cover-letter.html\n/home/hermes/.hermes/dashboard/templates/Anthony_Martin_Cover_Letter_Template.docx\n\n\nThe old upload-cache location is not authoritative and may be evicted.\nRequired nonempty artifacts: .txt, .json, .html, .docx, .pdf.\nThe HTML must contain zero unresolved {{...}} tokens and return HTTP 200.\nRetry generation once when an artifact, Notion link or HTTP check fails.\nA clean run requires MISSING_COVER_LETTER_COUNT=0. Any remaining missing letter must be reported prominently with its Notion page ID; it must never be silently omitted.\n\nLive audit — 27 July 2026\n\nSame-day strong matches: 1\nVerified canonical HTML cover letters: 1\nMissing cover letters: 0\nThe Market Creations Agency strong match has all five artifact formats, zero unresolved template tokens and a live HTTP 200 HTML link.\n\nRequired production pipeline\n\nObtain the live AWST date and calculate a seven-day freshness cutoff.\nSearch SEEK, LinkedIn, Indeed, WA Government Jobs and direct employer sites.\nOpen and verify every individual advert. Search snippets are discovery evidence only.\nReject stale, expired, cached, interstate, non-marketing, salary-guide, search/category and aggregator pages.\nReject all Pacific Energy roles. Anthony was made redundant there in July 2026; old Pacific Energy adverts must never be presented as opportunities.\nAccept only direct individual advert URLs:\n\nSEEK /job/<numeric-id>\nLinkedIn /jobs/view/...<numeric-id>\nIndeed /viewjob?jk=<real-id>\ndirect employer career pages\nindividual WA Government role pages\n\n\nA WA Government slug page is valid without AdvertID in its URL when the page proves the exact role and agency, WA location, unique job/pool reference, future closing date and active Apply Now function.\nQuery the complete Notion database with live pagination. Deduplicate by exact URL, normalized company plus role, and materially similar company-role combinations.\nCreate only verified nonduplicates and require returned Notion page IDs.\nRefresh POST http://127.0.0.1:9099/api/refresh, then confirm every page and direct URL through /api/jobs.\nScore newly created roles against /home/hermes/.hermes/radar-ref/resumes/current_resume_from_website.txt using the documented 100-point rubric. Strong Match starts at 70.\nFor each new strong match, run:\n\n/home/hermes/.hermes/hermes-agent/venv/bin/python \\\n  /home/hermes/.hermes/scripts/regenerate_cover_letter.py \\\n  <NOTION_PAGE_ID> \\\n  "Generate the initial tailored cover letter using the verified job description. Preserve Anthony's factual employment history and use the canonical teal Montserrat template."\n\nVerify nonempty .txt, .json, .html, .docx and .pdf artifacts; no unresolved {{...}} tokens; and a live .html Notion Cover Letter URL.\nReport only facts produced by tools in that run. Never estimate counts or fabricate Notion writes.\n\nDashboard behaviour\nSource: /home/hermes/.hermes/dashboard/index.html\nBackend: /home/hermes/.hermes/scripts/job_dashboard_server.py\nLayout (top to bottom)\n\nHero header — RADAR logo, name, dateline\nMarket Intel panel — expandable salary/location/source stats from /api/market-intel\nKPI strip — clickable filter cards: Total Jobs, Backlog, Applied, Phone Screen, Interviews, Offers, Closing Soon\nFilter bar — text search, view toggles (table/cards), Refresh button\nStats strip — This Week, Stale 14d+, Strong Match, Past Close, Applied Rate\nPipeline Funnel — clickable per-status breakdown with bar chart\nJob table/cards — sortable columns, inline status pills, cover letter links, edit actions\n\nColumn sorting\nClick any column header (Status, Company, Role, Match, Salary, Age, Closes, Updated) to sort ascending; click again for descending. Arrow indicator shows current direction.\nStatus management\n\nBulk update: select jobs via checkboxes → pick status from dropdown → Apply. Sends {ids, updates: {status}} to POST /api/jobs/bulk-update.\nSingle edit: click ✎ → modal with Status, Priority, Role, Salary, Closing Date, URL, Notes, Cover Letter, Attachments. Saves to Notion via POST /api/jobs/<id>/update.\nFunnel/status filters: clicking a status in the funnel or KPI strip filters the table to that status.\n\nRich text notes\n\nWrite/Preview editor in the edit modal (monospace textarea + markdown preview)\nSupports: bold **, italic *, links [text](url), bullet lists - item\nSyncs to Notion Notes / JD Summary rich_text property on every save\n\nDocument attachments\n\nAttach any URL (documents, links, files) to a job entry\nStored as bookmark blocks on the Notion page body (not a property)\nLazy-loaded when edit modal opens: GET /api/jobs/<id>/attachments\nAdd: POST /api/jobs/<id>/attachments with {name, url}\nDelete: POST /api/jobs/<id>/attachments/delete with {block_id}\nVisible in Notion when you open the page\n\nCover letter integration\n\nRegenerate button (🔄) on each row calls POST /api/jobs/<id>/cover-letter/regenerate\nCL URL stored in Notion Cover Letter property and rendered as clickable link in table\nAll 5 artifact formats: TXT, JSON, HTML, DOCX, PDF\n\nMarket Intel\n\nExpandable panel at top of page\nData from GET /api/market-intel: total tracked, salary stats (avg/median/min/max), counts\nLazy-loaded on expand; also refreshes on ↺ Refresh\n\nAPI endpoints\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nEndpointMethodPurpose/api/jobsGETAll jobs (cached, paginated from Notion)/api/statsGETAggregate statistics/api/refreshPOSTForce Notion cache refresh/api/market-intelGETSalary/location/source analytics/api/jobs/<id>/updatePOSTUpdate single job properties/api/jobs/<id>/attachmentsGETFetch page block attachments/api/jobs/<id>/attachmentsPOSTAdd bookmark attachment to page/api/jobs/<id>/attachments/deletePOSTRemove block from page/api/jobs/<id>/cover-letter/regeneratePOSTRegenerate cover letter/api/jobs/<id>/matchGETMatch analysis for a job/api/jobs/bulk-updatePOSTUpdate multiple jobs at once/api/jobs/createPOSTCreate new job in Notion\n\nCompany names and job titles with a valid URL render as blue external links and open the direct advert in a new tab.\nURL values are protocol-validated; only http: and https: are rendered as links.\nKPI/stat cards are one-click filters with an active highlight:\n\nTotal Jobs\nBacklog\nApplied\nInterviews\nOffers\nClosing Soon\nThis Week\nStale\nStrong Matches\nPast Closing\nApplication Rate\n\n\nFunnel status rows are also one-click exact-status filters.\nExisting status editing remains through the job edit modal and bulk-action controls, which write back to Notion through /api/update.\nThe frontend filter/link test harness is /tmp/radar_dashboard_test.js (temporary; recreate if absent).\n\n27 July 2026 incident\nSymptoms\n\nThe morning cron returned old and irrelevant listings, including the stale Pacific Energy Marketing Specialist advert associated with Anthony’s redundancy.\nAggregator mirrors, generic SEEK salary pages and category pages were treated as job adverts.\nCounts and quality signals were presented without a demonstrated live Notion query/write trail.\nThe dashboard had valid URLs for almost every record, but the frontend never rendered job.url; users could not open the actual adverts.\nThe earlier prompt explicitly narrowed the task so far that it accidentally prohibited the documented strong-match cover-letter stage.\n\nRecovery\n\nArchived all 16 records created by the contaminated morning run: 16 successful, zero failures.\nReplaced the cron prompt with deterministic freshness, source, verification, Notion, scoring, exclusion and cover-letter gates.\nPinned the cron primary to gpt-5.6-terra.\nAdded and directly probed opencode-go / mimo-v2.5; it returned MIMO_FALLBACK_OK, then was inserted first in the inherited fallback chain.\nRepaired regenerate_cover_letter.py:\n\nabsolute Hermes paths rather than ~ expansion\none correct command dispatcher rather than two conflicting __main__ blocks\n\n\nRepaired dashboard direct links and one-click quick filters.\nThe corrected Terra run created four verified fresh direct LinkedIn roles. One scored 75 and produced all five cover-letter artifact formats.\nA follow-up run queried 166 live Notion records across two pages, verified three current direct adverts, and correctly skipped all three as duplicates. No Pacific Energy records were added.\n\nFinal verification\n\nDashboard API: 166 records\nRecords created on 27 July after cleanup: 4\nPacific Energy records created on 27 July: 0\nRecords with direct URLs: 164 of 166\nStrong-match cover letter artifact sizes were nonzero for all five formats\nUnresolved template tokens: 0\nDashboard, API and generated cover-letter URLs returned HTTP 200\nSynthetic dashboard tests passed all quick-filter categories, unsafe-URL rejection and rendered job-link checks\n\nTailscale route recovery found during verification\nThe dashboard’s local service was healthy while the tailnet URL failed because the CT460 :443 Serve table had been cleared. Running the original atomic helper restored Serve but also erased the public :8443 Funnel route.\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh now atomically restores both:\n\ntailnet-only :443: /, /jobs, /desktop, /vnc-camofox, /evening-grid.html\npublic :8443: /webhook → http://127.0.0.1:8085/webhook\n\nImportant: tailscale serve reset clears Funnel state too. Never restore only the 443 routes and assume 8443 survived.\nVerification commands\n# Cron configuration\nhermes cron list\n \n# Fallback order\nhermes fallback list\n \n# Dashboard/API\ncurl -I hermes.kangaroo-eel.ts.net/jobs/\ncurl -sS hermes.kangaroo-eel.ts.net/jobs/api/jobs | jq '.jobs | length'\n \n# Local service\nss -ltnp 'sport = :9099'\ncurl -sS http://127.0.0.1:9099/api/jobs | jq '.jobs | length'\n \n# Route table\ntailscale serve status\n \n# Atomic route rebuild\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run\n/home/hermes/.hermes/scripts/tailscale-serve-apply.sh\nBackups from the incident\n\n/home/hermes/workspace/index.html.radar-links-backup-20260727\n/home/hermes/workspace/jobs.json.radar-prompt-backup-20260727\n/home/hermes/workspace/config.yaml.pre-mimo-fallback-20260727\n\nRelated\n\nscheduled-tasks\ntailscale\nmodel-providers\n"},"runbooks/multiplexer-setup":{"slug":"runbooks/multiplexer-setup","filePath":"runbooks/multiplexer-setup.md","title":"Multiplexer Setup (Multi-Profile Gateway)","links":["systems/hermes-agent","systems/messaging-integrations"],"tags":["hermes","multiplexer","telegram","profiles","gateway"],"content":"Multiplexer Setup\nWhat It Is\nOne gateway process serving multiple Hermes profiles through a single Telegram bot. Messages are routed to the correct profile by profile_routes (chat_id → profile).\nArchitecture\nTelegram Bot (single token)\n         │\n         ▼\n   Default Gateway (multiplex_profiles: true)\n         │\n         ├── profile_routes: chat_id → profile\n         │\n         ├── default profile (Rhino chat: -1004321904721)\n         └── ops profile (Tusk ops: -1003914987043, -1003932503629)\n\nKey Config Locations\nDefault profile: ~/.hermes/config.yaml\ngateway:\n  multiplex_profiles: true\n  profile_routes:\n    - name: ops-group\n      platform: telegram\n      chat_id: '-1003914987043'\n      profile: ops\n    - name: ops-group-2\n      platform: telegram\n      chat_id: '-1003932503629'\n      profile: ops\n \ntelegram:\n  allowed_chats: '-1004321904721,-1003914987043,-1003932503629'\n  group_allowed_chats: '-1004321904721,-1003914987043,-1003932503629'\nOps profile: ~/.hermes/profiles/ops/config.yaml\ntelegram:\n  allowed_chats:\n    - '-1003914987043'\n    - '-1003932503629'\n  group_allowed_chats:\n    - '-1003914987043'\n    - '-1003932503629'\n \nplatforms:\n  telegram:\n    enabled: false   # ← DISABLED — multiplexer handles this\nThe Silent-Drop Pitfall\nThe allowed_chats gate runs on the default profile’s adapter BEFORE profile_routes stamps source.profile. If a chat is only authorized in the secondary profile’s config but not the default’s, messages from that chat are silently dropped.\nMoral: When adding a new chat for a secondary profile, add it to BOTH profiles’ allowed_chats.\nAdding a New Group\n\nAdd the chat_id to the default profile’s profile_routes\nAdd the chat_id to the ops profile’s allowed_chats and group_allowed_chats\nAdd the chat_id to the default profile’s allowed_chats and group_allowed_chats (comma-separated list)\nRestart the gateway: systemctl --user restart hermes-gateway\n\nRestarting\nFrom an SSH shell (outside the agent process):\nsystemctl --user restart hermes-gateway\nRelated\n\nhermes-agent\nmessaging-integrations\n"},"runbooks/provider-health":{"slug":"runbooks/provider-health","filePath":"runbooks/provider-health.md","title":"Provider Health Check","links":["systems/model-providers","systems/messaging-integrations","runbooks/restart-hermes"],"tags":["runbook","providers","health","routing"],"content":"Provider Health Check\nPurpose\nVerify Hermes can reach LiteLLM, OmniRoute, OpenRouter-style fallbacks, and the WebUI, and report which paths are healthy or degraded.\nSymptoms that match this runbook\n\nModels list is slow or blank in Hermes\nFallback model retries happen immediately\nChat returns 5xx or gateway errors\n\nPrerequisites\n\nNetwork access from this host to litellm:4000, omniroute:20128, opencode.ai, api.telegram.org, Telegram fallback IPs\nHermes config slocated at ~/.hermes/config.yaml\n\nProcedure\n1. Check local Hermes services\nhermes gateway status\nhermes dashboard status\n2. Check main provider reachability\npython3 -c "import urllib.request; urllib.request.urlopen('http://litellm:4000/v1/models', timeout=4)"\nCurrent install behavior: LiteLLM responds but requires auth, so HTTP 401 is expected from an unauthenticated check.\n3. Check OmniRoute reachability\npython3 -c "\nimport urllib.request, json, yaml\nwith open('/home/hermes/.hermes/config.yaml') as f:\n    cfg = yaml.safe_load(f)\nkey = [p['api_key'] for p in cfg.get('custom_providers',[]) if p.get('name')=='omniroute'][0]\nreq = urllib.request.Request('http://omniroute:20128/v1/models', headers={'Authorization': f'Bearer {key}'})\nwith urllib.request.urlopen(req, timeout=5) as r:\n    data = json.loads(r.read())\n    print('OmniRoute OK, models:', len(data.get('data', [])))\n"\nCurrent verified result: reachable and returns a non-empty model catalog.\n4. Check fallback provider endpoints\ncurl -s -o /dev/null -w "%{http_code}" --max-time 5 opencode.ai/zen/v1/models\ncurl -s -o /dev/null -w "%{http_code}" --max-time 5 inference-api.nousresearch.com/v1/models\ncurl -s -o /dev/null -w "%{http_code}" --max-time 5 ollama.com/v1/models\nAny non-200 here means that fallback path is currently offline.\n5. Check WebUI health\ncurl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health\nExpected: 200\n6. Check gateway platform reachability\nTail the gateway journal for errors:\njournalctl --user -u hermes-gateway.service -n 100 --no-pager\nLook for repeated failures on Telegram IMAP, Telegram API, Slack Socket Mode, or Email fetch.\nVerification\n\nGateway status shows active (running).\nLiteLLM 401 from unauthenticated check is expected.\nOmniRoute returns model count > 0.\nFallback URLs return 200 or expected auth codes.\nWebUI /health returns 200.\nGateway journal shows no red alarm pattern.\n\nCurrent install references\n\nHermes: v0.19.0, git install: /home/hermes/.hermes/hermes-agent\nLiteLLM base URL: http://litellm:4000/v1\nOmniRoute base URL: http://omniroute:20128/v1\nOmniRoute direct IP: 100.88.81.19:20128\nWebUI: http://0.0.0.0:8787/health\n\nRollback\n\nThis read-only runbook has no rollback side effects\nIf you change fallback providers during remediation, record and revert via hermes fallback list + previous wiki state\n\nLast tested\n2026-07-22\nRelated\n\nmodel-providers\nmessaging-integrations\nrestart-hermes\n"},"runbooks/recover-docker-service":{"slug":"runbooks/recover-docker-service","filePath":"runbooks/recover-docker-service.md","title":"Recover a Docker service","links":[],"tags":["runbook","docker","recovery"],"content":"Recover Docker service\nProcedure\n\nIdentify the container.\nInspect status and last 100 log lines.\nIf unhealthy, restart then re-check.\nIf still failing, inspect image and mounts.\n\nVerification\nContainer shows healthy after recovery probe reboots.\nLast successfully used\n2026-07-22"},"runbooks/restart-browser":{"slug":"runbooks/restart-browser","filePath":"runbooks/restart-browser.md","title":"Restart Browser Helper","links":["systems/browser-backend","runbooks/restart-hermes"],"tags":["runbook","browser","cua-driver","camofox"],"content":"Restart Browser Helper\nPurpose\nRestart the computer-use browser stack that Hermes uses for web automation, screenshots, and VNC/noVNC access.\nSymptoms that match this runbook\n\nBrowser screenshots are failing\ncua-driver MCP tools return errors\nVNC/noVNC is unreachable on 5901 / 6081\n\nPrerequisites\n\nDisplay helper already running: Xvfb :99, x11vnc on 127.0.0.1:5901, noVNC at 127.0.0.1:6081\ncua-driver binary installed at /home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver\nBrowser VM/desktop helper is expected but not mandatory\n\nProcedure\n1. Inspect current browser helpers\nps aux | grep -E "cua-driver serve|x11vnc|fluxbox|vnc-watcher" | grep -v grep\n2. Stop cua-driver only\npkill -f "cua-driver serve"\n3. Start cua-driver\n/home/hermes/.local/bin/cua-driver serve\nRun in background or a tmux/screen session if needed.\n4. Verify ports remain available\nnc -z 127.0.0.1 5901 && echo "x11vnc ok" || echo "x11vnc missing"\ncurl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:6081/\n5. Restart full browser VM stack only if step 4 fails\npkill -f "Xvfb :99"\npkill -f "x11vnc"\npkill -f "fluxbox"\n/home/hermes/.local/bin/cua-driver serve\nVerification\n\ncua-driver serve process is present in ps\n5901 and 6081 are reachable\nNo zombie accumulation in browser/desktop processes\n\nRollback\n\nIf browser automation is not mission-critical, skip full VM restart and continue without it\nDo not stop gateway or dashboard unless necessary\n\nNotes\n\nThis runbook assumes the standalone desktop helper is running separately from Hermes core\nThe browser backend path can work without browser automation; this stack is optional depending on task\n\nLast tested\n2026-07-22\nRelated\n\nbrowser-backend\nrestart-hermes\n"},"runbooks/restart-hermes":{"slug":"runbooks/restart-hermes","filePath":"runbooks/restart-hermes.md","title":"Restart Hermes","links":["runbooks/backup-wiki","runbooks/update-hermes-safely"],"tags":["runbook","hermes","restart"],"content":"Restart Hermes\nPurpose\nRestart the Hermes agent, gateway, dashboard, and browser helper processes cleanly.\nSymptoms that match this runbook\n\nHermes stops responding in chat or WebUI\nTools fail mid-session but the agent is otherwise running\nAfter an update or config change\n\nPrerequisites\n\nUser account hermes with access to systemd —user\nNo active backups in progress\nIf using a browser session, warn the user before restarting dashboards\n\nProcedure\n1. Identify the active Hermes processes\nps aux | grep -E "hermes_cli.main (gateway|dashboard) run|hermes-webui/server.py|cua-driver serve|x11vnc.*5901" | grep -v grep\nExpected active pieces on this install:\n\ngateway: /home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run\ndashboard: /home/hermes/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main dashboard ...\nwebui: /home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py\ncamofox/cua-driver: /home/hermes/.local/bin/cua-driver serve\nx11vnc: -display :99 ... 5901\n\n2. Restart the gateway\nhermes gateway restart\n3. Restart the dashboard if needed\nhermes dashboard restart\nIf that fails, stop and rerun:\nhermes dashboard stop\nhermes dashboard --host 0.0.0.0 --port 9119 --no-open\n4. Restart the browser helper if needed\npkill -f "cua-driver serve"\n/home/hermes/.local/bin/cua-driver serve\n5. Restart the WebUI if needed\nFind its PID, then restart standalone:\npkill -f "hermes-webui/server.py"\n/home/hermes/.hermes/hermes-agent/venv/bin/python /home/hermes/hermes-webui/server.py\nVerification\nhermes gateway status\nhermes dashboard status\nss -tlnp | grep -E "8787|9119|5901|6081"\nps aux | grep -E "gateway run|dashboard|cua-driver serve|server.py" | grep -v grep\nCurrent known-good ports after verified restart:\n\ngateway: active via hermes-gateway.service\ndashboard: 0.0.0.0:9119\nwebui: 0.0.0.0:8787\nVNC: 5901\nnoVNC: 6081\n\nRollback\n\nIf restart fails, review recent config changes and consider restoring config.yaml.bak.*\nDo not delete old config files without explicit confirmation\nRe-run backup first if you plan to rollback config\n\nNotes\n\nLast tested: 2026-07-22 on debian 13, user-mode systemd\nWebUI logs: /home/hermes/.hermes/webui/bootstrap-8787.log\nDefunct <defunct> processes in process list are expected zombie shells from prior runs and do not require action unless they pile up\n\nRelated\n\nbackup-wiki\nupdate-hermes-safely\n"},"runbooks/restore-openrouter":{"slug":"runbooks/restore-openrouter","filePath":"runbooks/restore-openrouter.md","title":"Restore OpenRouter connectivity","links":[],"tags":["runbook","openrouter","recovery"],"content":"Restore OpenRouter connectivity\nSymptoms\n\nTLS connection hangs\nModel requests time out\nOther providers still work\n\nProcedure\n\nCheck session activity; ensure previous HTTPS calls completed.\nReview last WARP errors.\nRestart the Hermes process with the GoOp tool.\nSend a small test completion request.\nConfirm tool calling works.\n\nVerification\nA test prompt returns successfully through OpenRouter.\nEscalation\nIf the failure persists, check DNS, certificate expiry and provider status.\nLast successfully used\n2026-06-22"},"runbooks/test-openrouter":{"slug":"runbooks/test-openrouter","filePath":"runbooks/test-openrouter.md","title":"Test OpenRouter","links":["runbooks/provider-health","systems/model-providers"],"tags":["runbook","openrouter","providers","fallback"],"content":"Test OpenRouter\nPurpose\nTest whether OpenRouter is available to Hermes and whether fallback providers are actually callable in this installation.\nSymptoms that match this runbook\n\nHermes reports fallback model errors\nhermes fallback list shows expected providers, but actual requests fail\nUser wants to confirm OpenRouter is usable without relying on catalog names\n\nPrerequisites\n\nHermes CLI available: /home/hermes/.local/bin/hermes\nInternet egress for opencode.ai, inference-api.nousresearch.com, ollama.com, kilo.ai, litellm:4000\n\nProcedure\n1. List fallback providers\nhermes fallback list\nExpected active fallbacks on this install:\n\ndeepseek-v4-flash-free via opencode-zen\nstepfun/step-3.7-flash:free via nous\nqwen3.5:397b via ollama-cloud\npoolside/laguna-m.1:free via kilocode\nor-inclusionai_ling-2_6-1t via custom/litellm\n\n2. Test OpenRouter-style direct access if present\ncurl -s -o /dev/null -w "%{http_code}" --max-time 8 openrouter.ai/api/v1/models\nCurrent install state as of last run: OpenRouter is not configured as a standalone provider in Hermes, so this is an optional confirmation step only.\n3. Test each verified fallback URL\nfor url in \\\n  "opencode.ai/zen/v1/models" \\\n  "inference-api.nousresearch.com/v1/models" \\\n  "ollama.com/v1/models" \\\n  "api.kilo.ai/api/gateway/models"; do\n  printf "%s -> " "$url"\n  curl -s -o /dev/null -w "%{http_code}" --max-time 8 "$url"\n  echo\ndone\n4. Check Hermes fallback config reflects reality\ngrep -A5 "^fallback:" ~/.hermes/config.yaml\nVerify provider names and URLs in config match what the user wants to use.\nVerification\n\nhermes fallback list shows the exact same providers as the live config\nEach URL returns an HTTP response within timeout\nNo leftover undefined OpenRouter provider entries exist in custom_providers\n\nRollback\n\nNo system changes in this runbook\nIf a provider looks wrong but is part of live chat behavior, treat it like a config issue, not a transient health failure\n\nNotes\n\nIn this installation, OpenRouter is not a live first-class provider in custom_providers\nThis runbook tests confirmed fallback paths, not assumed ones\n\nLast tested\n2026-07-22\nRelated\n\nprovider-health\nmodel-providers\n"},"runbooks/troubleshoot-tool-loop":{"slug":"runbooks/troubleshoot-tool-loop","filePath":"runbooks/troubleshoot-tool-loop.md","title":"Troubleshoot Tool Loop","links":[],"tags":["runbook","hermes"],"content":"\nStub — structure only. Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.\n\nTroubleshoot Tool Loop\nSymptoms\nAgent loops on one tool or endpoint.\nProcedure\n\nInspect recent Hermes agent logs.\nIdentify the looping tool path.\nRestart worker or flush stuck request.\nRe-run limited test scenario.\n\nVerification\nAgent handles the same scenario without repeating.\nLast successfully used\n2026-07-22"},"runbooks/update-hermes-safely":{"slug":"runbooks/update-hermes-safely","filePath":"runbooks/update-hermes-safely.md","title":"Update Hermes safely","links":["runbooks/provider-health","runbooks/restart-hermes","runbooks/backup-wiki"],"tags":["runbook","hermes","update"],"content":"Update Hermes safely\nPurpose\nUpdate Hermes to a newer release without losing configuration, skills, or sessions.\nSymptoms that match this runbook\n\nUser wants to move to a newer Hermes version\nhermes version shows an older build than expected\nUpdate-related errors after manual changes in the install directory\n\nPrerequisites\n\nHermes install method: git-based install at /home/hermes/.hermes/hermes-agent\nWorking Python virtualenv at /home/hermes/.hermes/hermes-agent/venv\nAvailable disk space in Hermes install and backup paths\nNo active Hermes chat/webui shell issues before starting\n\nProcedure\n1. Create a backup first\nhermes backup --quick --label pre-update-$(date +%Y%m%d)\nIf you need a full backup instead:\nhermes backup --label pre-update-$(date +%Y%m%d)\n2. Check whether an update is available\nhermes update --check\n3. Review configuration version and backups\npython3 -c "import yaml; print(yaml.safe_load(open('/home/hermes/.hermes/config.yaml')).get('_config_version'))"\nls -la ~/.hermes/config.yaml.bak*\n4. Run the update with noninteractive flags\ncd /home/hermes/.hermes/hermes-agent\ngit status --short\nhermes update --yes --branch main\n5. Verify the new version\nhermes version\n6. Restart the required Hermes pieces\nhermes gateway status\nhermes dashboard status\nhermes gateway restart\nhermes dashboard restart\n7. Sanity check health\nhermes gateway status\ncurl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8787/health\nVerification\n\nhermes version shows a newer version than before\nhermes gateway status is active\nWebUI /health returns 200\nProviders reachable with provider-health\n\nRollback\n\nRollback Hermes config:\n\nls -lt ~/.hermes/config.yaml.bak*\ncp ~/.hermes/config.yaml.bak.<timestamp> ~/.hermes/config.yaml\nhermes gateway restart\nhermes dashboard restart\n\nRollback entire Hermes install requires git revert or reinstall from previous checkout\nIf update warns about config migration, revert the backup and re-run manually\n\nNotes\n\nHermes version on this install: v0.19.0 (2026.7.20)\nInstall directory: /home/hermes/.hermes/hermes-agent\nUpdate preserves venv by default unless --force-venv is used\n\nLast tested\n2026-07-22 on debian 13, git-based Hermes install\nRelated\n\nrestart-hermes\nbackup-wiki\nprovider-health\n"},"systems/browser-backend":{"slug":"systems/browser-backend","filePath":"systems/browser-backend.md","title":"Browser Backend","links":["comparisons/steel-vs-camoufox","concepts/browser-automation","decisions/rollback-steel-to-camofox","runbooks/restart-browser"],"tags":["browser","camofox","automation"],"content":"Browser Backend\nPurpose\nBrowser automation for Hermes — web search, form filling, screenshot, CDP control, authenticated scraping.\nCurrent Configuration (as of 2026-08-02)\n\nPrimary backend: Camoufox (Firefox-based, anti-fingerprint) on CT450\nProvider config: browser.cloud_provider: camofox, camera_provider: camofox\nREST API: localhost:9093 (profile anthony, sessionKey anthony)\nCDP WebSocket: localhost:9377 (via Camoufox’s Playwright bridge)\nVNC viewer: ct460.kangaroo-eel.ts.net/vnc-camofox (port 6080)\nSteel Browser (CT201): Still running but Hermes not connected — plugin disabled\n\nCamoufox (CT450)\n\nHost: CT450 (same host as Hermes — no network hop)\nDeployment: Systemd service camofox-browser.service\nEngine: Camoufox (custom Firefox build with anti-detection fingerprinting)\nDisplay: Headed via Xvfb :99 (1600×1200×24) + fluxbox window manager\nAuth: Persistent profile anthony — authenticated into Uber Eats, LinkedIn, etc.\nConnection: REST API at localhost:9093 for session management, CDP WebSocket at localhost:9377 for browser tools\nLast verified: Jul 30 — browserConnected: true, sessions: 1, tabs: 1, active 200s on scroll/navigate\n\nAccess\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nEndpointURLUsed byREST APIhttp://localhost:9093Hermes browser tools (navigate, click, snapshot)CDP WebSocketws://localhost:9377Direct CDP controlVNCct460.kangaroo-eel.ts.net/vnc-camofoxVisual session viewerCamoufox UIhttp://localhost:9378Internal management\nConfig\n\ncloud_provider: camofox in ~/.hermes/config.yaml\nCAMOFOX_URL=http://localhost:9093 in ~/.hermes/.env\nCAMOFOX_API_KEY in ~/.hermes/.env\nProfile: anthony (502913bf) — NOT persist-anthony\n\nSteel Browser (CT201) — Disabled\n\nStatus: Still running on CT201, Hermes not connected since 2026-07-30\nPlugin: browser-steel at /home/hermes/.hermes/plugins/browser-steel/ — disabled\nCDP: http://100.96.244.39:9223\nREST API: http://100.96.244.39:3000\nUI: runtipi.kangaroo-eel.ts.net/steel\nSee: steel-vs-camoufox for the full experiment story\n\nHistory\n\n2026-07-22: Camoufox primary, cua-driver bridge\n2026-07-29: Steel Browser deployed on CT201, became primary\n2026-07-30: Steel reverted — CDP issues + DeepSeek 0731 tool name validation problems. Camoufox restored as primary.\n2026-08-02: Camoufox confirmed working with authenticated sessions (Uber Eats report via Camofox REST API)\n\nRelated\n\nsteel-vs-camoufox\nbrowser-automation\nrollback-steel-to-camofox\nrestart-browser\n"},"systems/cloudflare-tunnel":{"slug":"systems/cloudflare-tunnel","filePath":"systems/cloudflare-tunnel.md","title":"Cloudflare Tunnel","links":["infrastructure/domains-and-tunnels","systems/hermes-agent","infrastructure/network-map","runbooks/diagnose-docker"],"tags":["system","network","cloudflare"],"content":"Cloudflare Tunnel\nPurpose\nExposes internal homelab services, including Hermes, to the public internet without opening inbound firewall ports.\nLocation\n\nRuns on: ct207 “webserver” (Tailscale 100.82.40.23), not on this host.\nBinary: /bin/cloudflared\nService: cloudflared.service (systemd, active/running on ct207)\nTunnel name / ID: ct207-tunnel / 339a6757-5574-4723-a16c-da4769535caa\nConfig: /etc/cloudflared/config.yml on ct207; credentials file 339a6757-5574-4723-a16c-da4769535caa.json in the same directory (not read/copied here — credential material, not a fact to mirror).\n\nCurrent configuration\n\noriginRequest: noTLSVerify: true, disableChunkedEncoding: true, http2Origin: false (global defaults; some hostnames override per-entry)\nIngress is a flat list of hostname → service mappings ending in service: http_status:404 catch-all — see domains-and-tunnels for the full list and the entries specific to Hermes.\nTwo Hermes-relevant entries point at this host’s Tailscale IP (100.118.5.51): hermes.martinwa.org (port 19119) and hermes-desktop.martinwa.org (port 5174).\n\nConnected systems\n\nhermes-agent — the service actually reached through this tunnel\ndomains-and-tunnels — full hostname table\nnetwork-map\n\nKnown issues\n\nThis host cannot restart or inspect the tunnel directly (no cloudflared installed here); a tunnel-side problem requires access to ct207.\nmcp-portal.martinwa.org, referenced in Hermes’s own config as an MCP endpoint, was not found in the ct207 ingress list captured 2026-07-22 — unresolved, may be on a different tunnel/host.\n\nRelevant runbooks\n\ndiagnose-docker (closest existing runbook; no dedicated tunnel-restart runbook exists yet — worth writing one that documents systemctl restart cloudflared on ct207, since Hermes has no direct access to perform it)\n"},"systems/current-profile":{"slug":"systems/current-profile","filePath":"systems/current-profile.md","title":"Current Hermes Profile","links":["systems/hermes-agent","systems/model-providers"],"tags":["system","hermes","profile"],"content":"Current Hermes Profile\nActive Profile\n\nName: default\nLocation: ~/.hermes/profiles/default/\nNo other profiles confirmed\n\nKey Settings\n\nModel default: freellmapi-auto\nDefault provider: custom:litellm → http://litellm:4000/v1\nSwitch persistence: persist_switch_by_default: false\nTimezone: Australia/Perth\nApprovals mode: off\nMemory: Mnemosyne, session-scoped, auto-sleep enabled\nPet slug: none\nSkin: slate\n\nPaths\n\nConfig: ~/.hermes/config.yaml\nWorking directory: /home/hermes/workspace\nInstall: ~/.hermes/hermes-agent/\nBin: ~/.local/bin/hermes\n\nRelated\n\nhermes-agent\nmodel-providers\n"},"systems/guanaco":{"slug":"systems/guanaco","filePath":"systems/guanaco.md","title":"Guanaco","links":["infrastructure/proxmox","systems/ollama"],"tags":["llm","proxy","router","guanaco","ollama"],"content":"Guanaco\nPurpose\nLLM proxy/router providing model routing and tool-call formatting for the Ollama-based inference stack on CT205.\nCurrent Configuration\n\nHost: CT205 (ollama LXC container) on Proxmox\nVersion: v0.8.14 (updated from v0.8.11 on 2026-08-14)\nRepo path (on CT205): /root/.guanaco/repo\nAPI endpoint: ollama:11434 (DNS) / localhost:11434\nKey component: cmdcode_client.py — Guanaco’s tool-call formatting layer (DSML)\n\nVersion History\nv0.8.14 (current, updated 2026-08-14)\n\nDSML tag leakage fixes (cumulative across v0.8.12–v0.8.14):\n\nv0.8.12 — incomplete DSML blocks leaking raw tags as content\nv0.8.13 — comprehensive DSML tag leakage: double-pipe, bare fragments, solo invoke\nv0.8.14 — <dsml_ignore> blocks, bare parameter values, stray > cleanup\n\n\nDashboard performance fix (analytics.py): fixed wrong column names in optimized history query, cached content size, stopped SELECT * on history table\nNo new features, no breaking config changes — low-risk incremental bug-fix release\nLocal app.py modification (working-tree change) survived the tag checkout intact\n\nv0.8.11 (previous)\n\nHad known DSML tag leakage issues where malformed tool-call output could leak raw <dsml_...> syntax into model context stream\n\nWhat DSML Is\nDSML = “Daily Smart Messaging Log” — Guanaco’s tool-call formatting layer for the cmdcode client. When a model emits a tool call, Guanaco formats it through DSML. Malformed output from the model could previously leak raw tags (<dsml_invoke>, bare > characters) into the conversation context, which confused downstream models.\nRelated\n\nproxmox — CT205 in LXC container table\nollama — Ollama inference stack (if exists)\n"},"systems/headless-server":{"slug":"systems/headless-server","filePath":"systems/headless-server.md","title":"Headless Server","links":["systems/hermes-agent","systems/browser-backend","comparisons/camoufox-vs-chrome","infrastructure/hosts","runbooks/restart-browser"],"tags":["system","browser"],"content":"Headless Server\nPurpose\nHeadless X11 environment backing Hermes’s browser automation and any GUI tooling.\nLocation\n\nHost: this container (ct460, hermes), no separate VM/LXC — headless stack runs directly alongside the agent.\nDisplay: Xvfb :99, resolution 1600x1200x24, started with -ac -nolisten tcp\n\nCurrent configuration (verified via ps aux, 2026-07-22)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProcessDetailXvfb:99 -screen 0 1600x1200x24 -ac -nolisten tcpfluxboxwindow manager on :99x11vnc-display :99 -forever -shared -rfbport 5901 -rfbportv6 5901 -noxdamage -quiet -nopw — no VNC password setwebsockify ×2127.0.0.1:6080 → 127.0.0.1:5900 and 127.0.0.1:6081 → 127.0.0.1:5901, serving noVNC (/usr/share/novnc)\nTwo VNC/websockify pairs exist (5900/6080 and 5901/6081) — the second (:99→5901→6081) is the one confirmed above; the first (5900→6080) predates it and wasn’t independently re-verified this pass — unresolved whether it’s a second live display or a leftover process.\nConnected systems\n\nhermes-agent (browser backend uses this display — see browser-backend / camoufox-vs-chrome)\nhosts\n\nKnown issues\n\nx11vnc -nopw: VNC is unauthenticated. It’s bound to all interfaces (0.0.0.0:5901 and [::]:5901), not just loopback — reachable from the LAN and, if any tunnel/firewall rule forwards it, potentially beyond. Worth restricting to loopback or adding -passwd/-rfbauth if this hasn’t been a deliberate choice.\n\nRelevant runbooks\n\nrestart-browser\n"},"systems/hermes-agent":{"slug":"systems/hermes-agent","filePath":"systems/hermes-agent.md","title":"Hermes Agent","links":["systems/omniroute","systems/model-providers","systems/memory-backend","systems/browser-backend","systems/terminal-backend","systems/skills-index","runbooks/multiplexer-setup","systems/messaging-integrations","systems/scheduled-tasks","current-state"],"tags":["system","hermes","agent","llm"],"content":"Hermes Agent\nPurpose\nPrimary personal agent for Anthony. Runs automation, research, communication, knowledge management, browser control, and scheduled tasks.\nCurrent Status\n\nVersion: v0.19.0 (2026.7.20), upstream 9ecacd6b\nInstall method: git\nPython: 3.11.15\nOpenAI SDK: 2.24.0\nUpstream status: Up to date\nInstall directory: /home/hermes/.hermes/hermes-agent\nCommunity resources: Hermes Registry — third-party plugins, skills, and extensions\nBinary: /home/hermes/.local/bin/hermes\nConfig: ~/.hermes/config.yaml (version 33)\nProfile: default (~/.hermes/profiles/default/)\n\nLocation\n\nHost: Local Linux host (no containerized Hermes runtime confirmed)\nFilesystem root: ~/.hermes/\nWorking directory: /home/hermes/workspace\n\nProviders & Routing\n\nDefault provider: custom:litellm → http://litellm:4000/v1\nFallback provider: custom:omniroute → http://omniroute:20128/v1\nDefault model: freellmapi-auto\nProvider model count: ~3430 (omniroute), ~1512 (litellm)\nfree-stack combo (OmniRoute): Nemotron 3 Ultra 550B configured with max_tokens: 8192, reasoning_effort: medium, chat_template_kwargs.force_nonempty_content: true (see omniroute)\nSee model-providers for full breakdown.\n\nMemory\n\nBackend: Mnemosyne (local native)\nMemory enabled: true\nChar limit: 5000\nUser profile enabled: true\nUser char limit: 3000\nFlush every: 6 turns\nAuto sleep: true\nDefault scope: session\nVector type: int8\nShared surface: data/shared/mnemosyne.db\nSee memory-backend for details.\n\nBrowser\n\nCloud provider: Camoufox\nManaged persistence: true\nInactivity timeout: 120 s\nAllow private URLs: true\ncua-driver: v0.9.0 at /home/hermes/.cua-driver/packages/releases/0.9.0-x86_64-unknown-linux-gnu/cua-driver\nDISPLAY: :102 (Xvfb/X11VNC/fluxbox confirmed working per session notes)\nSee browser-backend for details.\n\nTerminal\n\nBackend: local\nWorking directory: /home/hermes/workspace\nTimeout: 180 s\nHome mode: auto\nContainer CPU: 1\nPersistent shell: true\nSee terminal-backend for details.\n\nSkills\n\nTotal installed: 66\nKey skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.\nSee skills-index for categorized list.\n\nMulti-Profile Multiplexer\nThe default gateway runs as a profile multiplexer (gateway.multiplex_profiles: true), serving both the default and ops profiles through a single gateway process with one Telegram connection.\nProfiles\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProfileChatPurposedefault-1004321904721 (Rhino chat)Main agentops-1003914987043 (Elephant ops), -1003932503629 (OPS Tember rescue)Operations/infra agent\nHow It Works\n\nOne gateway, one Telegram bot token — shared across profiles\nprofile_routes maps chat_id → profile at the gateway level\nallowed_chats on both profiles must include ALL chats that either profile should hear, because the default profile’s adapter processes all inbound messages before profile_routes stamps source.profile\nThe ops profile does NOT need platforms.telegram.enabled — the default adapter handles it\n\nCritical Rule: allowed_chats Is Pre-Route\nThe allowed_chats gate runs on the default profile’s adapter before profile_routes matching. If a chat_id is only in the ops profile’s allowed_chats but not the default’s, messages from that chat are silently dropped before routing ever fires.\nFix applied 2026-07-28: Added both ops group IDs to the default profile’s allowed_chats and group_allowed_chats (comma-separated). See multiplexer-setup.\nMessaging Integrations\n\nTelegram: enabled, allowed_chats: -1004321904721,-1003914987043,-1003932503629, rich messages\nDiscord: enabled, streaming true\nSlack: enabled\nEmail: connected\nMattermost: connected\nntfy: connected\nGoogle Chat: available (hermes-google_chat)\nQQ: available (hermes-qqbot)\nYuanbao: available\nSee messaging-integrations for details.\n\nScheduled Tasks\n\nCron jobs: 25 active jobs\nDelivery targets: Telegram (primary), Discord, origin, local\nNotable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher\nSee scheduled-tasks for full table.\n\nDependencies\n\nLiteLLM proxy at litellm:4000\nOmniRoute at omniroute:20128\ncua-driver binary at path above\nXvfb + X11VNC + fluxbox for headed browser\nTelegram bot token, Discord token, etc. (stored outside wiki)\n\nKnown Issues\n\nRouter migration incomplete\nBrowser persistence not yet fully verified\nObsidian Git plugin auto-sync caused conflicts during vault reorganization\n\nRelated\n\nmodel-providers\nmemory-backend\nbrowser-backend\nterminal-backend\nskills-index\nmessaging-integrations\nscheduled-tasks\ncurrent-state\n"},"systems/hermes-ui":{"slug":"systems/hermes-ui","filePath":"systems/hermes-ui.md","title":"hermes-ui (Browser Hermes Desktop)","links":[],"tags":["hermes","webui","desktop","browser"],"content":"hermes-ui\nThe official Hermes Desktop renderer, extracted from the hermes-agent monorepo (apps/desktop) and repackaged as a plain Vite web app — runs in a browser instead of Electron.\n\nRepo: github.com/przbadu/hermes-ui\nUpstream: NousResearch/hermes-agent monorepo, commit 56a8e81d33a524f0ba0d68b6d54c8786ed283fb8 (2026-07-08)\nExtraction date: 2026-07-11\nLicense: MIT (Copyright (c) 2025 Nous Research)\n\nDeployments\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nHostLocationPortStatusStartedCT460 (this box)/home/hermes/hermes-ui/5174Vite dev serverActiveCT450 (hermesagent)/home/hermes/hermes-ui/5174Vite dev serverActive (since 2026-07-30)\nHow It Works\n\nRuns as a Vite dev server, proxying REST/auth/WebSocket to a running Hermes gateway (http://127.0.0.1:9119 by default).\nSame-origin proxy via Vite so cookies and the WebSocket work without CORS issues.\nElectron-only methods are stubbed behind a capability flag in src/web-bridge/.\n\nQuick Start\n# Install deps (uses bun)\ncd ~/hermes-ui/app && bun install\n \n# Start dev server\ncd ~/hermes-ui && bin/dev\n \n# Or skip bin/dev and run vite directly:\nbunx vite --port 5174 --host 0.0.0.0\nFiles\n\nPLAN.md — extraction plan\nUPSTREAM.md — upstream commit diff guide\nscripts/serve-on-gateway.sh — serve built UI same-origin via hermes serve\n"},"systems/home-assistant":{"slug":"systems/home-assistant","filePath":"systems/home-assistant.md","title":"Home Assistant","links":["systems/hermes-agent","infrastructure/proxmox","systems/messaging-integrations","systems/homepage-dashboard"],"tags":["system","automation"],"content":"Home Assistant\nPurpose\nHome automation and device control; one of Hermes’s messaging/integration platforms.\nLocation\n\nVerified (homelab infra docs, not this host): VM 100 on pve, hostname haos14.0 (Home Assistant OS), internal IP 172.30.232.1.\nVerified (Hermes config): homeassistant and hermes-homeassistant appear in Hermes’s platform/display integration lists (~/.hermes/config.yaml), alongside qqbot, signal, slack, teams, telegram.\nUnknown: the actual connection details (HA base URL, long-lived access token, which entities/automations Hermes can reach) are not present in the sanitized config snapshot — likely held in a plugin-specific credentials file not captured there, or not yet wired up beyond being enabled as a platform.\n\nCurrent configuration\nNot verified beyond the platform being listed as available/connected in current-state.md’s “WebUI connected platforms” line. No automations, entity lists, or voice-assistant config confirmed from this host.\nConnected systems\n\nhermes-agent\nproxmox — Runs as VM 100 on PVE\nmessaging-integrations — Integration platform\nhomepage-dashboard — Shown in dashboard More tab\n\nKnown issues\n\nGenuine gap, not a “known issue” per se: this page cannot confirm HA is actively used by Hermes vs. merely enabled — worth verifying directly (e.g. a live call to a known HA entity) next time someone is in this area.\n\nRelevant runbooks\nNone yet — would be a good candidate once the connection is verified (e.g. runbooks/restart-home-assistant-bridge)."},"systems/homepage-dashboard":{"slug":"systems/homepage-dashboard","filePath":"systems/homepage-dashboard.md","title":"Homepage Dashboard","links":[],"tags":["dashboard","homepage","homelab","proxmox","monitoring"],"content":"Homepage Dashboard\nPurpose\nCentral dashboard for all homelab services, with Proxmox integration, per-VM stats, and a management UI.\nURLs\n\nTailscale: runtipi.kangaroo-eel.ts.net:8443/\nLocal: runtipi.kangaroo-eel.ts.net:8082/\nManager UI: runtipi.kangaroo-eel.ts.net:8084/\n\nArchitecture\nCT201 (runtipi)\n├── Homepage container (port 8082) — main dashboard\n├── Manager container (port 8084) — web UI for adding/removing apps\n└── Tailscale serve (port 8443) — HTTPS proxy to Homepage\n\nTabs\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTabContentOverviewInfrastructure (2 cols) + Everyday (4 cols) + ResourcesMediaJellyfin, arr stack, RSSAIOmniRoute, LiteLLM, Lobe Chat, SearXNGAgentsHermes, OpenClaw, Nanobot, CamofoxToolsIT tools, Cyber Chef, etc. + StorageMoreHome Assistant, N8n, JupyterAll Apps114 apps in 5-column grid (alphabetical)\nProxmox Integration\n\nNode widget: Shows live VM/LXC counts, CPU, memory for PVE node\nPer-VM stats: Click “View Proxmox stats” on mapped services\nAPI token: homepage@pve!homepage (privsep=0)\nproxmox.yaml: Config at /opt/homepage/config/proxmox.yaml\n\nVM/LXC Mapping\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nServiceVMIDTypePVE(node)—Portainer docker200lxcPortainer debianVM101qemuPortainer openweb245lxc\n\nPortainer on CT245 (openweb.kangaroo-eel.ts.net:9443) now uses a Tailscale/Let’s Encrypt cert (installed 2026-08-09). Files: /root/portainer-certs/portainer.{crt,key} mounted into the container, --sslcert/--sslkey. Renewal: /root/renew-portainer-cert.sh via crontab (04:00 on the 1st monthly) — tailscale cert --min-validity=20d then docker restart portainer. Cert validity ~90 days.\n| Runtipi | 201 | lxc |\n| Gitea | 221 | lxc |\n| FreshRSS | 260 | lxc |\n| NextCloud | 270 | lxc |\n| LiteLLM | 500 | lxc |\n| OmniRoute | 600 | lxc |\n| Ntfy | 241 | lxc |\n| Home Assistant | 100 | qemu |\n| N8n | 215 | lxc |\n| Mattermost | 255 | lxc |\n| SFTPGo | 299 | lxc |\n| Cliproxy API | 103 | lxc |\n| OpenClaw | 403 | qemu |\n| Zeroclaw | 333 | lxc |\n| Hermes | 460 | lxc |\n| Hindsight | 450 | lxc |\n\nConfiguration Files\nAll at /opt/homepage/config/ on CT201:\n\nservices.yaml — All apps, groups, and Proxmox mappings\nsettings.yaml — Theme, layout, tabs\ncustom.css — Warm amber accent overrides\nwidgets.yaml — System health, clock\nproxmox.yaml — PVE API token\ndocker.yaml — Docker socket integration\n\nManager UI\nWeb form at port 8084 for adding/removing apps without SSH.\n\nAdd: Name, Category (dropdown), URL, optional Icon\nDelete: Search apps, click Delete\nAuto-restart: Homepage restarts after each change\n\nQuick Launch Search\nType anywhere on the dashboard to search services by name/description. Google search disabled — only searches your apps.\nTheme\n\nDark base with warm amber (#d4a04a) accents\nBigger tiles (68px min-height, 40px icons)\nCustom CSS in custom.css\n\nTailscale Serve\nPort 443 occupied by Traefik on CT201 — Tailscale uses port 8443 instead.\nMaintenance\n\nRestart Homepage: docker restart homepage\nRestart Manager: docker restart homepage-manager\nEdit apps manually: Edit /opt/homepage/config/services.yaml then restart\nDocker auto-discovery: Add homepage.group, homepage.name, homepage.href labels to containers\n"},"systems/memory-backend":{"slug":"systems/memory-backend","filePath":"systems/memory-backend.md","title":"Memory Backend","links":["incidents/2026-08-01-mnemosyne-capture-failure","comparisons/memory-backends","systems/memory-backend","systems/mnemosyne","experiments/"],"tags":["memory","mnemosyne","agent"],"content":"Memory Backend\nPurpose\nPersist and recall agent memory across sessions.\nCurrent Configuration\n\nPrimary backend: Mnemosyne (local native)\nMemory enabled: true\nUser profile enabled: true\nMemory char limit: 5000\nUser char limit: 3000\nFlush every: 6 turns\nAuto sleep: true\nDefault scope: session\nVector type: int8\nProfile isolation: true (configured; requires gateway restart for the current messaging runtime)\nShared surface: data/shared/mnemosyne.db\nShared surface read: true\nSync roles: user\nSkip contexts: cron, flush, subagent, background, skill_loop\nReflect max calls per session: 3\n\nStorage Location\n\nMnemosyne home: ~/.hermes/mnemosyne/\nDB: ~/.hermes/mnemosyne/data/\nBackups: ~/.hermes/mnemosyne/backups/\nLogs: ~/.hermes/mnemosyne/logs/\nModels: ~/.hermes/mnemosyne/models/\nConfig: ~/.hermes/mnemosyne/config.yaml\n\nKnown Issues\n\nHealth monitoring currently treats capture as stale only after 24 hours; an activity-aware comparison is preferable to catch dry-intake failures sooner without false alarms during idle periods.\nThe 2026-08-01 automatic-capture failure and repair are documented in 2026-08-01-mnemosyne-capture-failure.\n\nCorrection (2026-07-22, per Anthony): this page previously referenced Hindsight as a possibly-used backend on this box. That was a mix-up with a different Hermes instance — see memory-backends.\nRelated\n\nmemory-backend\nmnemosyne\nindex\n"},"systems/messaging-integrations":{"slug":"systems/messaging-integrations","filePath":"systems/messaging-integrations.md","title":"Messaging Integrations","links":["systems/messaging-integrations","systems/scheduled-tasks"],"tags":["messaging","telegram","discord","email","integrations"],"content":"Messaging Integrations\nTelegram\n\nStatus: enabled, active\nallowed_chats: -1004321904721 (primary)\nRich messages: true\nPlatform toolsets: cli, telegram (full suite)\nDelivery targets seen in cron: telegram, telegram:1793951355, telegram:-1004321904721:1030, telegram:-1004321907D2:346, telegram:-1004321904721:273\nNotes: Primary delivery target for scheduled jobs\n\nDiscord\n\nStatus: enabled, streaming true\nPlatform: hermes-discord\nPlatform toolsets: cli, discord\n\nEmail\n\nStatus: connected\nLibraries: himalaya CLI available\nPlatform toolsets: cli\n\nSlack\n\nStatus: available (toolsets: cli, slack-platform)\nConfig noted: slack entries in config but Slack enabled status unclear from config grep\n\nOther Platforms Available\n\nMattermost: toolsets present (hermes-mattermost)\nGoogle Chat: toolsets present (hermes-google_chat)\nQQ: toolsets present (hermes-qqbot)\nSignal: toolsets present (hermes-signal)\nWhatsApp: toolsets present (hermes-whatsapp)\nntfy: noted in memory config outputs but runtime status unverified\nYuanbao: toolsets present (hermes-yuanbao)\nHome Assistant: homeassistant-platform enabled\n\nKnown Issues\n\nTelegram thread_id 34795 for telegram:1793951355 not found — one cron job delivered without thread_id\n\nRelated\n\nmessaging-integrations\nscheduled-tasks\n"},"systems/mnemosyne":{"slug":"systems/mnemosyne","filePath":"systems/mnemosyne.md","title":"Mnemosyne","links":["incidents/2026-08-01-mnemosyne-capture-failure","incidents/2026-07-29-mnemosyne-relay","concepts/agent-memory","comparisons/memory-backends"],"tags":["memory","mnemosyne","backend"],"content":"Mnemosyne\nPurpose\nPrimary agent memory store for Hermes.\nCurrent Configuration\n\nProvider: mnemosyne\nEnabled: true\nAuto sleep: true\nDefault scope: session\nMemory char limit: 5000\nUser char limit: 3000\nVector type: int8\nProfile isolation: true\nShared surface read: true\nShared surface path: data/shared/mnemosyne.db\nSkip contexts: cron, flush, subagent, background, skill_loop\nReflect max calls per session: 3\nSync roles: user\nFlush every: 6 turns\nEmbedding model: BAAI/bge-small-en-v1.5 (384 dims)\nEmbeddings via API: false (local fastembed)\n\nDatabase Stats (2026-08-02)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTableRowsNotesworking_memory1,476Active memoriesepisodic_memory458Consolidated summariesmemory_embeddings158Only working memories — 0 episodicmemory (legacy)2,181Old table, not migratedvec_working159Vector index rowsShared surface0Empty — cross-agent sharing not functional\nKnown Issues\nVector embedding coverage gap\n\nOnly 159/1,476 working memories (10.8%) have vector embeddings\n0/458 episodic memories have embeddings\nSemantic search (mnemosyne_recall) falls back to FTS5 keyword matching for ~90% of working and 100% of episodic memories\nThis means recall quality is degraded — vector similarity can only search a tiny fraction of the memory store\nRoot cause: The embedding pipeline (fastembed) never processed the bulk of memories. The vec_working table was rebuilt once (Jul 18) but only captured 159 rows.\nFix needed: Backfill embeddings for all working and episodic memories. This requires running the embedding pipeline across the full dataset.\n\nProfile isolation chosen over shared surface\n|- The ops profile was enabled with mnemosyne on 2026-08-02\n|- Decision: profile_isolation: true was set on ops, giving each profile its own independent memory bank — zero cross-contamination\n|- data/shared/mnemosyne.db exists with 0 rows; cross-profile sharing is intentionally unused\n|- This means Tusk (ops) and Rhino (default) do not share memories, which is the desired behaviour\n|- Config in ~/.hermes/profiles/ops/config.yaml: memory.mnemosyne.profile_isolation: true, allow_tool_override: false, shared_surface_read: false\n|- Verified end-to-end on 2026-08-02: memory writes land correctly in both profiles independently\nLegacy table not migrated\n\n2,181 rows in the old memory table are not in the new working_memory/episodic_memory system\nThese are only accessible via the legacy memory tool, not via mnemosyne_recall\n\nKnown Management Jobs\n\nMnemosyne Auto-Consolidate every 30m — script: consolidate_mnemosyne.sh, mode: no-agent\nMnemosyne DB Backup weekly Sunday 03:00 — script: mnemosyne-backup.sh, mode: no-agent\nMnemosyne Health Monitor every 12h — skill: mnemosyne-operations, reports only on exception\nMnemosyne Watchdog every 15m — script: mnemosyne-watchdog.py, mode: no-agent\n\nHistory and Evidence\n\n2026-08-01 incident: Automatic conversational capture was broken by stringified structured YAML, especially sync_roles: "['user']". Fixed by rewriting with native YAML types. See 2026-08-01-mnemosyne-capture-failure.\n2026-08-01 cleanup: Legacy always-injected memory reduced from 8,013 bytes to 4,998 bytes (memory) and 2,958 bytes (user).\n2026-07-29 incident: memory.mnemosyne.tools: None caused schema loading warnings. Fixed by removing the line. See 2026-07-29-mnemosyne-relay.\n2026-07-29 relay scope corruption: RuntimeError: scope handle is not at the top of the stack — 3 occurrences, self-resolved.\n\nRelated\n\nagent-memory\nmemory-backends\n2026-08-01-mnemosyne-capture-failure\n2026-07-29-mnemosyne-relay\n"},"systems/model-providers":{"slug":"systems/model-providers","filePath":"systems/model-providers.md","title":"Model Providers","links":["systems/guanaco","incidents/2026-08-04-omniroute-free-deepseek-silent-stops","concepts/model-routing","systems/omniroute","systems/opencode-go"],"tags":["model","provider","inference","routing"],"content":"Model Providers\nPurpose\nInference providers available to Hermes, their routing paths, and notable model updates.\nCurrent Provider Stack\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nProviderURLRolecustom:litellmhttp://litellm:4000/v1Main inference (slower path, 6–26s/call)custom:omnihttp://omni:20128/v1Auxiliary roles (vision, skills, approval, mcp, web_extract, compression) — factory-reset 2026-08-14opencode-zen(internal)Free-tier model access for cron jobsGuanacoollama:11434LLM proxy/router on CT205 (Ollama stack)\nDefault Model\n\nresilient-free — resolves through OmniRoute (changed from auto/best-free on 2026-08-14 after factory reset)\nPrevious default auto/best-free was misconfigured — had no combo definition and routed to paid gpt-5.6-sol\nfreellmapi-auto (LiteLLM path) still available but slower\n\nNotable Model Updates\nGPT-5.6 Luna — Model Research (2026-08-14)\n\nCommunity consensus: Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper\nLuna strengths: visual/UI work, final code review, bounded high-stakes tasks, cleaner structured output\nDeepSeek strengths: automation, persistent executor work, high-volume subtasks, long-context economics\nKey caveat: Provider route matters more than model name — openai-codex/gpt-5.6-luna, openrouter/openai/gpt-5.6-luna, deepseek/deepseek-v4-flash, and openrouter/deepseek/deepseek-v4-flash are four different tests\nPricing: Direct OpenAI 0.20/M input, 1.20/M output; OpenRouter 0.10/M input, 0.60/M output\nContext: 1.05M tokens, 128K output, function calling, structured outputs, image input, configurable reasoning effort\nRecommendation: Luna plans and reviews; DeepSeek executes — but test on a clean direct route before judging\n\nGuanaco v0.8.14 (2026-08-14)\n\nUpdated from v0.8.11 on CT205 — three incremental bug-fix releases focused on DSML/tool-call output hygiene\nDSML tag leakage fixes: malformed tool-call output no longer leaks raw syntax junk into model context\nDashboard performance fix: cached content size, fixed wrong column names in history query\nNo new features, no breaking changes — low-risk update\nSee guanaco\n\nDeepSeek V4 Flash 0731 (2026-07-31)\n\nSame model ID: deepseek-v4-flash — no ID change, just a post-training bump\nSame architecture: 284B total params, 13B active per token, 1M context window\nSame pricing: 0.14/M input, 0.28/M output\nBenchmark jump: Terminal Bench 2.1: 82.7 (up from 61.8 preview — 34% relative gain)\nNow surpasses V4-Pro preview (72.1) on agent/coding benchmarks\nStricter tool name validation: Rejects tool names not matching ^[a-zA-Z0-9_-]+$ — may cause issues with OmniRoute-transformed tool names\nImpact: Your existing cron jobs using deepseek-v4-flash get the upgrade automatically. The empty-stream fallback issue (opencode-go/deepseek-v4-flash returning empty 200s) is a provider plumbing problem, not a model quality problem.\n\nOmniRoute Free-DeepSeek Silent-Stop Mitigation (2026-08-04)\n\nFree-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances\nReworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort\nSee 2026-08-04-omniroute-free-deepseek-silent-stops\n\nContext Enforcement\n\nThree layers: model native → combo data.config.context_length → Hermes client cap\nEffective context is the minimum of the three\n\nCombo Behavior\n\npriority retries same provider up to maxRetries, then errors; it does NOT fall through\nfill-first walks the list on failure\nauto/lkgp recalculates per request using health scoring\nSame-provider entries in a combo share rate-limit buckets\nSession affinity caches provider in session_model_history\n\nRelated\n\nmodel-routing\nomniroute\nopencode-go\n"},"systems/nanobot":{"slug":"systems/nanobot","filePath":"systems/nanobot.md","title":"Nanobot","links":["infrastructure/domains-and-tunnels","comparisons/hermes-vs-nanobot","systems/hermes-agent","systems/headless-server","runbooks/restart-hermes"],"tags":["system","nanobot","network"],"content":"Nanobot\nPurpose\nRole in Anthony’s setup.\nLocation\n\nVerified 2026-07-22: LXC 333, hostname clawtest, Tailscale IP 100.122.189.81.\n\nCurrent state\n\nStatus: DOWN (as of 2026-08-07). Connection refused on port 8900 — the OmniRoute provider connection to nanobot (58debd59, prefix nanobot) was causing event-loop stalls because every model sync + credential health check retried it. Connection disabled in OmniRoute. Nanobot on CT333 is not responding.\nPrior state: nanobot process listening on port 8766 (verified via ss -tlnp on ct333, 2026-07-22).\nwhatsapp-bridge.service (systemd, active) — “WhatsApp Bridge for Nanobot” — also runs on ct333. Status unknown as of 2026-08-07.\nPublicly reachable via the same Cloudflare Tunnel as Hermes: nanobot.martinwa.org → 100.122.189.81:8766, nano-ssh.martinwa.org → SSH on the same host. See domains-and-tunnels.\n\nCurrent state as of 2026-07-22 (superseded — see below)\n- No nanobot process running on this host\n- nanobot systemd user service: inactive\n- Not present in pgrep process list\nCorrection (this audit, same day): the above was true only for this host (ct460) — Nanobot was never expected to run here. It is actively running, on LXC 333 (“clawtest”), confirmed via direct process/port check. Left the original lines struck through rather than deleted, per SCHEMA rule 9 (mark, don’t silently delete).\nHistory\n\nUser-provided: Nanobox on the PVE host at 192.168.178.39\nCorrection: 192.168.178.39 is pve’s own LAN IP (matches the Cloudflare Tunnel’s pve.martinwa.org entry, which proxies to Proxmox’s web UI on that address) — this was a conflation, not a second Nanobot instance. The real Nanobot host is ct333 (see Location above).\nVerified: Lan/Wi-Fi node at 192.168.178.46 running ARM/arm64 firmware; ubnt login at 192.168.1.1 does not apply to this node\nVerified: DWARF01 identified as a UniFi device; ONT01 status unresolved; DNS provided by 192.168.178.39, not the router\nUser-provided: Avoid heroku.com/archive except for CAPI; use official CAPI course downloads instead\nSuperseded: “Nanobot may have been used previously as Hermes gateway or tunnel front on another host” / “Unknown: whether nanobot should be redeployed” — no longer applicable now that it’s confirmed running; the open question is now why it’s running alongside Hermes and whether that’s intentional (see hermes-vs-nanobot).\n\nConnected systems\n\nhermes-agent\nheadless-server\n\nKnown issues\n\nOverlap/purpose vs. Hermes not documented anywhere — see hermes-vs-nanobot for the open question this raises.\n\nChange history\n\n2026-07-22: audit corrected “not running” claim after direct verification on ct333.\n2026-08-07: status changed to DEGRADED — nanobot confirmed DOWN (connection refused on 8900). OmniRoute provider connection disabled. CT333 service status unknown.\n\nRelevant runbooks\n\nrestart-hermes\n\nRelated\n\nhermes-vs-nanobot\n"},"systems/obsidian":{"slug":"systems/obsidian","filePath":"systems/obsidian.md","title":"Obsidian Vault","links":["infrastructure/hosts","systems/memory-backend"],"tags":["obsidian","notes","vault","homelab"],"content":"Obsidian Vault\nPurpose\nPersonal knowledge base — notes, research, projects, and agent documentation.\nDetails\n\nLocation: /home/hermes/wiki/ (on CT460)\nGit repo: Gitea (Anthony/obsidian-vault)\nStructure: NUMBERED folders (00-99)\nSync: Git-based (manual or cron-triggered)\n\nFolder Structure\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nFolderPurpose00 - HomeDashboard and entry points10 - InboxNew/unprocessed notes20 - Agent NotesNotes from AI agents30 - ProjectsActive project documentation50 - ResearchResearch and investigations60 - ResourcesReference materials70 - ReferenceQuick lookup90 - ArchiveCompleted/old material99 - TemplatesNote templates\nConventions\n\nNumbered folders preferred (Anthony corrected twice)\nCheck Mnemosyne before vault work\nDon’t impose structure changes without understanding usage\n\nRelated\n\nhosts — CT460 host details\nmemory-backend — Agent memory integration\n"},"systems/omniroute":{"slug":"systems/omniroute","filePath":"systems/omniroute.md","title":"OmniRoute","links":["incidents/2026-08-04-omniroute-free-deepseek-silent-stops","systems/model-providers","concepts/model-routing"],"tags":["routing","provider","model","inference"],"content":"OmniRoute\nPurpose\nSecondary provider/router for Hermes auxiliary roles and some agent routing.\nCurrent Configuration\n\nHermes provider name: custom:omni\nBase URL: http://omni:20128/v1\nHost reachability: omni.kangaroo-eel.ts.net resolves to 100.93.204.4; CT601 running, direct HTTP reachable\nRuns on: Proxmox CT601 (omni) — migrated from CT600 (omniroute, now stopped) on 2026-08-13\nDB path: /opt/omniroute/storage.sqlite (was /var/lib/omniroute/storage.sqlite)\nBackup: pre-migration config saved at ~/.hermes/config.yaml.bak.omniroute-removal\nMiss: Hermes default provider path does not use OmniRoute for main chat; used for auxiliary roles instead\nAuxiliary roles in Hermes config: skills_hub, approval, mcp\nKey prefix visible in config: sk-5c67fbdc0... — full value not stored in wiki\n\nCatalog and Routing Facts\n\nCatalog reachable: yes, 47 combos returned at /api/combos\nCombo schema observed: name, strategy, models, id; some combos have config, isHidden, uuid\nCombo-level context can override individual models\nauto/lkgp strategies may silently change chosen models\nModel mappings can substitute models silently\nSession affinity caches last successful provider in session_model_history\n\nCatalog Evidence (2026-07-22)\n\nAdvertised context for gemini/gemini-3.6-flash: 1048576\nRelated models in catalog: kc/google/gemini-3.6-flash, kilocode/google/gemini-3.6-flash, nous/google/gemini-3.6-flash, opencode-zen/gemini-3.6-flash, vertex/gemini-3.6-flash\nNo live combo in the returned 47 combos referenced gemini-3.6-flash\n\nKnown Issues\n\nopencode-zen proxy pitfall documented in omniroute-ops skill\nWeb search proxy failures documented in omniroute-ops skill\nCombo discovery can differ from resolution;\n\nAutoRoute variants (2026-08-12)\n\nauto/best-free (Auto Cheap) and auto/coding-free are virtual AutoRoute variants — they do NOT appear in the persisted combo list (/api/combos returns 54 combos, none named these).\nThe LKGP engine resolves them dynamically: it scores the operator’s live connected provider accounts (e.g. antigravity, opencode-zen, nvidia, openrouter) rather than a hardcoded candidate list.\nScoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.\nExample observed: an auto/best-free request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.\n\nRecent Changes (2026-08-14)\n\nOmniRoute factory reset on CT601. Database wiped clean: 0 combos, 0 providers, 0 API keys, 115 built-in auto model entries (auto/best-fast, auto/best-coding, etc.). Old state backed up at /opt/omniroute/factory-reset-backup-20260814-225524.\nProviders re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter — all five passed live provider tests. OpenAI keys in the vault were tested but none worked; removed the invalid OpenAI entry.\nHermes config default model changed: auto/best-free → resilient-free (via ~/.hermes/config.yaml). auto/best-free was misconfigured — had no combo definition and routed to gpt-5.6-sol (paid) instead of free models. Switched to resilient-free which is a working free combo.\nMultiple free combos broken by upstream model ID changes:\n\nOpenRouter: poolside/laguna-s-2-1:free (dash) → poolside/laguna-s-2.1:free (dot) — model ID format changed\nGroq: meta-llama/llama-4-scout-17b-16e-instruct — retired, only llama-3.3/3.1 remain\nAffected combos: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free\nAll route through OpenRouter → fail with 400/404 when trying these model IDs\n\n\nWorking free combos as of 2026-08-14: free-lkgp (DeepSeek V4 Flash), mega-free (StepFun 3.7 Flash), smart-route / smart-route-ia (nested combos)\nLiteLLM routing issue identified: Hermes was previously pointing at http://litellm:4000/v1 with free-auto model, causing 6–26s per call. OmniRoute’s auto/best-free tested at 265ms. Config now points at OmniRoute.\nGPT-5.6 Luna research: Community consensus — Luna is cleaner/more polished; DeepSeek V4 Flash is more persistent/cheaper. Luna better for visual/UI work and final review; DeepSeek better for automation and high-volume executor work. Provider route matters more than model name.\n\nRecent Changes (2026-08-13)\n\nOmniRoute migrated CT600 → CT601. Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute.kangaroo-eel.ts.net (100.88.81.19, offline). CT600 stopped, CT601 running. Database path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite. API key unchanged. All 14 config + 22 cron-job custom:omniroute references updated to custom:omni. Pre-migration backup at ~/.hermes/config.yaml.bak.omniroute-removal. Verified live 2026-08-13 (pct list: 600 stopped, 601 running; grep: 0 remaining custom:omniroute refs).\n\nRecent Changes (2026-07-25)\n\nfree-stack combo updated: Nemotron 3 Ultra 550B model at index 4 now configured with NVIDIA-specific parameters:\n\nmax_tokens: 8192 (was default 65536, exceeding NVIDIA NIM limit of 16384)\nreasoning_effort: medium (was high, reducing long internal monologues)\nchat_template_kwargs.force_nonempty_content: true (prevents empty responses when tool calling + reasoning)\n\n\nCombo ID: 23d25f8f-5eb0-4b6f-b966-e260ca3905b3\nHermes config.yaml updated: agent.reasoning_effort: medium, agent.max_tokens: 8192, agent.chat_template_kwargs.force_nonempty_content: true\nRationale: NVIDIA NIM API documents max output of 16384 tokens; medium reasoning uses more efficient chat-template mode; force_nonempty_content resolves empty responses with tool calls combo in picker may still route\n\nRecent Changes (2026-08-07)\n\nThree-layer root cause for slowness identified and fixed:\n\nnpm package .env override: OmniRoute loads three .env files (/var/lib/omniroute/.env, /root/.omniroute/.env, /usr/lib/node_modules/omniroute/.env). The npm package’s .env ships with ENABLE_SOCKS5_PROXY=true and npm update re-installs it with that default, silently overriding the /var/lib config. Fix: flip all three to false, then restart. After ANY upgrade, check all three.\nDead provider connection: Connection 58debd59 (prefix nanobot) points at clawtest:8900/v1 — nanobot on clawtest is DOWN. Every model sync + credential health check retried it, burning event-loop time. Disabled it.\nPVE host load leaking into CT600: CT600 has nesting=1, so uptime/top inside the container mirror PVE’s host load exactly (verified: identical 10.30/9.23/10.37 on both). Same trap as CT241. Judge CT600 CPU from pct exec 600 -- ps -o %cpu or host-side ps, never from in-container load average.\n\n\nFirst-request-after-cache-expiry stall: /v1/models hangs 10-15s+ on the FIRST hit after the model catalog cache expires (rebuild enumerates all connections, including slow dead ones), then answers in ~0.02s from cache. Don’t judge health on one curl — run a burst of 5 and check if later tries are sub-second.\nis_active=0 does NOT stop CredentialHealth probes: Disabling a dead connection stops it from ModelSync but the health checker still probes it and logs [ProxyFetch] ECONNREFUSED noise. That noise is benign once the endpoint is unreachable.\nLessons banked in omniroute-ops skill: references/omniroute-upgrade-procedure.md updated with all three pitfalls.\n\nRecent Changes (2026-08-04)\n\nOmniRoute free-DeepSeek silent-stop mitigation: Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See 2026-08-04-omniroute-free-deepseek-silent-stops.\n\nRelated\n\nmodel-providers\nmodel-routing\n"},"systems/opencode-go":{"slug":"systems/opencode-go","filePath":"systems/opencode-go.md","title":"OpenCode Go","links":[],"tags":["opencode","provider","models","subscription","limits"],"content":"OpenCode Go\nLow cost subscription for open coding models — 5 first month, then **10/month**. Designed for international users, models hosted in US, EU, and Singapore.\nHow It Works\n\nSign in to OpenCode Zen, subscribe to Go, copy your API key\nRun /connect in the TUI, select OpenCode Go, paste your API key\nRun /models to see available models\n\n\nOnly one member per workspace can subscribe.\n\nUsage Limits\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWindowDollar Limit5 hours$12Weekly$30Monthly$60\nLimits are in dollar value, so your actual request count depends on the model. Cheaper models (DeepSeek V4 Flash, MiMo-V2.5) get far more requests than expensive ones (Kimi K3, Grok 4.5).\nEstimated Requests Per Window\nBased on typical Go usage patterns:\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nModelPer 5hPer WeekPer MonthGrok 4.5120300600GLM-5.28802,1504,300GLM-5.18802,1504,300Kimi K3110250490Kimi K2.7 Code1,3503,3806,750Kimi K2.61,1502,8805,750MiMo-V2.530,10075,200150,400MiMo-V2.5-Pro3,2508,15016,300MiniMax M33,2008,00016,000MiniMax M2.73,4008,50017,000Qwen3.7 Max9502,3904,770Qwen3.7 Plus4,30010,80021,600Qwen3.6 Plus3,3008,20016,300DeepSeek V4 Pro3,4508,55017,150DeepSeek V4 Flash31,65079,050158,150Hy34,30010,75021,500\nToken Estimates Per Request\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nModelInput TokensCached TokensOutput TokensGrok 4.51,10071,500220GLM-5.2/5.170052,000150Kimi K31,05076,500300Kimi K2.7/K2.687055,000200DeepSeek V4 Pro75082,000290DeepSeek V4 Flash79068,000280MiniMax M351056,000190MiniMax M2.730055,000125MiMo-V2.583071,500295MiMo-V2.5-Pro79086,000305Qwen3.7 Max42066,000200Qwen3.7 Plus50057,000190Qwen3.6 Plus50057,000190Hy383071,500295\nPer-Model Pricing & Monthly Usage Included\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nModelInput $/1MOutput $/1MCached Read $/1MMonthly UsageGrok 4.5$2.00$6.00$0.30$15GLM-5.2$1.40$4.40$0.26$60GLM-5.1$1.40$4.40$0.26$60Kimi K3$3.00$15.00$0.30$15Kimi K2.7 Code$0.95$4.00$0.19$60Kimi K2.6$0.95$4.00$0.16$60MiMo V2.5$0.14$0.28$0.0028$60MiMo V2.5 Pro$0.435$0.87$0.0036$15MiniMax M3$0.30$1.20$0.06$60MiniMax M2.7$0.30$1.20$0.06$60Qwen3.7 Max$2.50$7.50$0.50$60Qwen3.7 Plus (≤256K)$0.40$1.60$0.04$60Qwen3.7 Plus (>256K)$1.20$4.80$0.12$60Qwen3.6 Plus (≤256K)$0.50$3.00$0.05$60Qwen3.6 Plus (>256K)$2.00$6.00$0.20$60DeepSeek V4 Pro$0.435$0.87$0.0036$15DeepSeek V4 Flash$0.14$0.28$0.0028$60Hy3$0.14$0.58$0.035$60\nTrack usage in the console.\n\nIf you reach the usage limit, you can continue using the free models.\n\nUsage Beyond Limits\nIf you also have credits on your Zen balance, enable the Use balance option in the console. Go will fall back to your Zen balance after reaching usage limits instead of blocking requests.\nWhy Some Models Have Lower Usage\nWith Go, you pay 10/month and they aim to give you **6x that in usage** (60). For most models, this works through bulk discounts and reserved GPU capacity. For some models (Grok 4.5, Kimi K3, MiMo V2.5 Pro, DeepSeek V4 Pro), they haven’t negotiated discounts yet — so the usage multiplier is lower (you get a little more than paying directly, just not the full 6x).\nAvailable Models\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nModelModel IDEndpointSDK PackageGrok 4.5grok-4.5opencode.ai/zen/go/v1/chat/completions@ai-sdk/openai-compatibleGLM-5.2glm-5.2samesameGLM-5.1glm-5.1samesameKimi K3kimi-k3samesameKimi K2.7 Codekimi-k2.7-codesamesameKimi K2.6kimi-k2.6samesameDeepSeek V4 Prodeepseek-v4-prosamesameDeepSeek V4 Flashdeepseek-v4-flashsamesameMiMo-V2.5mimo-v2.5samesameMiMo-V2.5-Promimo-v2.5-prosamesameMiniMax M3minimax-m3opencode.ai/zen/go/v1/messages@ai-sdk/anthropicMiniMax M2.7minimax-m2.7samesameMiniMax M2.5minimax-m2.5samesameQwen3.7 Maxqwen3.7-maxsamesameQwen3.7 Plusqwen3.7-plussamesameQwen3.6 Plusqwen3.6-plussamesameHy3hy3opencode.ai/zen/go/v1/chat/completions@ai-sdk/openai-compatible\nConfig format: opencode-go/<model-id> (e.g. opencode-go/kimi-k3)\nModel list API: opencode.ai/zen/go/v1/models\nPrivacy\nZero-retention policy from providers — your data is not used for model training.\nGoals\n\nMake AI coding accessible via low cost subscription\nProvide reliable access to the best open coding models\nCurate tested and benchmarked models for coding agent use\nNo lock-in (use any other provider too)\n"},"systems/openrouter":{"slug":"systems/openrouter","filePath":"systems/openrouter.md","title":"OpenRouter","links":["systems/hermes-agent","systems/omniroute","runbooks/restore-openrouter"],"tags":["system","openrouter","provider"],"content":"OpenRouter\nPurpose\nFallback model provider.\nLocation\nRemote service; API token store.\nCurrent configuration\nPrimary key, fallback models, rate limits.\nCurrent state as of 2026-07-22\nOpenRouter is not configured as a standalone provider in ~/.hermes/config.yaml custom_providers.\nOmniRoute’s catalog includes provider names and model IDs consistent with OpenRouter-style slugs, but live routing via OpenRouter directly is not present in current Hermes config.\nHistory\n\nUser-provided: Previously used; now largely replaced by LiteLLM and OmniRoute paths\nVerified: Decision page keep-openrouter-as-fallback exists, indicating retain intent\nInferred: Multiple backups before model/routing changes imply OpenRouter role changed during prior config iterations\nUnknown: Exact date OpenRouter was replaced or partially deprecated; exact models that were available\n\nConnected systems\n\nhermes-agent\nomniroute\n\nKnown issues\n\nReplaced: legacy Hermes autos not persistent, prompting Hermes Agent, which blocks attribution.\n\nChange history\n\nPre-2026-07-20: OpenRouter provisioning active in some form\n2026-07-20: Prior model-fix backup suggests routing changes around this time\n\nRelevant runbooks\n\nrestore-openrouter\n"},"systems/scheduled-tasks":{"slug":"systems/scheduled-tasks","filePath":"systems/scheduled-tasks.md","title":"Scheduled Tasks","links":["runbooks/job-radar","current-state","systems/mnemosyne"],"tags":["cron","scheduled","automation"],"content":"Scheduled Tasks\nCount\n\nActive jobs: 28 (verified from live hermes cron list, 2026-08-07)\nPreviously 25 — GitHub Trending Radar Daily removed 2026-08-02; several jobs added since.\n\nJob Table\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nNameScheduleDeliverySkillsScriptNotesFacebook Birthday Monthly Scrape0 9 1 * *telegram——⚠ thread_id not foundFacebook Birthday Weekly Registry Sync0 3 * * 1telegram—birthday_check.py⚠ thread_id not foundDaily Birthday Checker30 7 * * *telegram—birthday_check.pyno-agentDaily Marketing Job Radar0 7 * * 1-5telegram:-1004321904721:1030job-search-automation—Primary gpt-5.6-terra; see job-radarPerth New Restaurant & Cafe Openings (Weekly)0 18 * * 4telegram——AI Newsletter Curation for Patrick (Weekly)0 10 * * 4telegram:-1004321904721:346——Enhanced Weekend Adventure Planner0 8 * * 5telegram——Tech & AI Newsletter Digest (Analytical)0 7 * * *telegram:-1004321904721:346——Political News Digest (Mon/Thu)0 8 * * 1,4telegram:-1004321904721:346——Uber Eats Monthly Spend Report0 2 1 * *telegramcamofox-browser-automation, browser-backend-verification—Gmail Action Digest30 8 * * *telegram:-1004321904721:notifications—gmail_action_digest_v3.pyFreshRSS Midday News Briefing0 12 * * *telegram:-1004321904721:346—freshrss_intelligence_v2.pyWorkday start reminder0 9 * * 1-5telegram——Important-mail monitorevery 30mtelegram:1793951355——Silent when no important mailMnemosyne Auto-Consolidateevery 30mlocal—consolidate_mnemosyne.shno-agentMnemosyne DB Backup0 3 * * 0telegram:1793951355—mnemosyne-backup.shno-agentMnemosyne Health Monitor0 */12 * * *telegram:-1004321904721:102mnemosyne-operations—Reports only on exceptionMnemosyne Watchdog*/15 * * * *telegram:-1004321904721:102—mnemosyne-watchdog.pyno-agent, silent ≤10 conflictsMnemosyne Embedding Coverage Monitor0 */6 * * *telegram:-1004321904721:102—check_embedding_coverage.pyAlerts if coverage < 95%CT 221 Miner Watchdogevery 15mtelegram:1793951355pve-lxc-intrusion-detectionct221-miner-watchdog.shAdded 2026-08-07 after crypto miner incidentMosquito Tracker0 7,15,22 * * *telegram:-1004321904721:273—mosquito_tracker.pyExcludes 23:00–07:00Weekly Review0 18 * * 0originweekly-review—Evening Stand-Down Briefing45 18 * * 1-5telegram:-1004321904721—evening_standdown.pyno-agentJob Tracker Dashboard Refresh0 */2 * * *telegram:-1004321904721:1030—dashboard_watchdog.pyno-agentWiki Audit — Nightly59 23 * * *localtask-approach—Reviews past 24h for wiki changesGitHub Trending Radar — Weekly Roundup0 9 * * 1telegram:1793951355——🦝 Hermes Config Backup to Gitea (Daily)0 11 * * *origin—hermes-config-backup.shno-agentReview Agent Estate Phase Oneonce at 2026-08-08 18:33origin——✅ Completed 2026-08-08. See current-state.\nRemoved Jobs\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nNameRemovedReasonGitHub Trending Radar — Daily Digest2026-08-02Stopped by user. Weekly roundup still active.\nKnown Issues\n\nSession storage failures (2026-08-10): Four cron jobs failed — two with TimeoutError (idle 601–602s while stuck initializing), two with RuntimeError (“session storage could not be written”). Disk at 83% (11GB free), state.db at 593MB. Likely WAL checkpoint or disk pressure issue. Affected jobs: Daily Marketing Job Radar, Tech & AI Newsletter Digest, Political News Digest, GitHub Trending Radar — Weekly Roundup. See current-state for details.\nWiki Audit cron model drift (2026-08-07): Global inference config changed (smart-route → deepseek-paid-smart) and the job is unpinned, causing RuntimeError: Skipped to prevent unintended spend. Needs pinning: cronjob action=update job_id=cbfe434a3b5e provider=<provider> model=<model>.\nFacebook Birthday thread_id warnings: Configured thread_id 34795 for telegram:1793951355 was not found on several birthday-related jobs — delivered without thread.\nModel repinning (2026-08-08): 13 cron jobs repinned from dead/stale provider combos to smart-route or gpt-5.6-terra. Dead combos included freeapi/minimax-m2.7, freeapi/auto, vertex/gemini-2.5-flash, deepseek-v4-flash-free. Jobs now running on valid model pins.\n\nRelated\n\nmnemosyne\njob-radar\n"},"systems/skills-index":{"slug":"systems/skills-index","filePath":"systems/skills-index.md","title":"Installed Skills","links":["systems/tool-search","systems/hermes-agent","systems/messaging-integrations"],"tags":["skills","hermes"],"content":"Installed Skills\nInventory\nTotal installed: 66 skills in ~/.hermes/skills/.\nCategorized Active Skills\nCore Hermes\n\nhermes — Core agent skill\n\nProviders & Routing\n\nomniroute — OmniRoute routing\nomniroute-ops — OmniRoute diagnostics and combo management\n\nMemory\n\nchromadb — ChromaDB vector store\nchromadb-preflight — ChromaDB RAG preflight\nchromadb-skills-rag — Skills RAG via ChromaDB\n\nBrowser / RPA\n\ncomputer-use — Desktop GUI automation\n\nDevOps / Homelab\n\ndevops — Umbrella for service operations\ndevops-homelab-architecture-live-probe — Live homelab diagram via probes\ndevops-proxmox-operations — Proxmox VE REST/SSH operations\ndevops-service-management — Miscellaneous service ops\ndevops-tailscale-web-routing — Tailscale web routing\ncloudflare-tunnel-ops — Cloudflare Tunnels\ntailscale-serve-ops — Tailscale Serve\ntailscale-ops — Tailscale configuration\n\nMessaging\n\nemail — Email from terminal\nemail-himalaya — Himalaya CLI IMAP/SMTP\n\nJob / Career\n\njob-hunting — Career transition coaching\njob-hunting-assistance — End-to-end job hunting support\njob-search-automation — Autonomous job board scraper\njob-tracker-enrichment — Notion job tracker enrichment\n\nContent / Research\n\ntech-ai-newsletter-digest — Newsletter processing\nmedia — YouTube transcripts, GIF search, music generation\n\nProductivity\n\nproductivity — Document creation, presentations, spreadsheets\nmarkdown-to-anywhere — Cross-posting service\nweekly-review — Sunday weekly review cron\n\nMCP / Skills Management\n\nmcp — MCP server tooling\nmcp-agent-integration — Local stdio MCP server wiring\n\nUnverified / Not Categorized\n\n40+ skills not inspected in detail during this inventory\nFull list at ~/.hermes/skills/\n\nHow category discovery works\nSkill categories are derived from the directory layout, not from Tool Search.\nThe first directory below ~/.hermes/skills/ becomes the category:\n\n~/.hermes/skills/devops/my-skill/SKILL.md → category devops\n~/.hermes/skills/my-skill/SKILL.md → no category\n\nThe skill’s display name and discovery description come from the YAML\nfrontmatter in SKILL.md. Directory additions are detected automatically.\nIn-place edits may take up to 30 seconds to appear because the discovery cache\nhas a 30-second lifetime.\nSee tool-search for the separate external-tool discovery system.\nRelated\n\nhermes-agent\nmessaging-integrations\ntool-search\n"},"systems/terminal-backend":{"slug":"systems/terminal-backend","filePath":"systems/terminal-backend.md","title":"Terminal Backend","links":["concepts/persistent-shells","systems/headless-server"],"tags":["terminal","shell","backend","persistence"],"content":"Terminal Backend\nPurpose\nExecute shell commands, scripts, and interactive sessions for Hermes.\nCurrent Configuration\n\nBackend: local\nWorking directory: /home/hermes/workspace\nTimeout: 180 s\nHome mode: auto\nContainer CPU: 1\nPersistent shell: true\n\nPersistence\n\nShell state (venv, env vars, aliases) persists across terminal calls\nWorking directory fixed to /home/hermes/workspace\nNo containerized terminal confirmed\n\nKnown Issues\n\nNone observed\n\nRelated\n\npersistent-shells\nheadless-server\n"},"systems/tool-search":{"slug":"systems/tool-search","filePath":"systems/tool-search.md","title":"Hermes Tool Search","links":["systems/hermes-agent","systems/skills-index","concepts/tool-calling","runbooks/gateway-resource-resilience"],"tags":["hermes","automation","system"],"content":"Hermes Tool Search\nWhy it is enabled\nHermes previously sent every external tool schema to the model on every turn.\nThat used a large part of the context window before the conversation began and\nmade compaction happen much sooner.\nNative Hermes Tool Search is now always on. Core tools remain immediately\navailable. External tools are discovered only when they are relevant, then\ndescribed and called through the Tool Search bridge.\nActive configuration\nFile: /home/hermes/.hermes/config.yaml\ntools:\n  tool_search:\n    enabled: "on"\n    threshold_pct: 10\n    search_default_limit: 5\n    max_search_limit: 20\nenabled: "on" is the important setting: it makes Tool Search active on every\nrequest instead of waiting for the tool schemas to cross an automatic token\nthreshold.\nVerified result\n\n80 external tools were deferred in the production gateway.\nApproximately 24,151 schema tokens were removed from the always-present\nprompt.\n36 core or directly visible tools remained available.\nEstimated fixed baseline fell from about 52,955 to 29,182 tokens, a reduction\nof roughly 45%.\nIn an isolated Computer Use test, 53 tools were discovered and the\nhealth_report search, description, and call all succeeded.\nNative Tool Search tests passed: 39 tests.\n\nTool Search can add a small discovery call when an external tool is first\nneeded. It should not reduce answer quality: the model retains direct access to\ncore tools and can search for specialized tools by name or purpose.\nNew tools and new skills\nNew external tools are automatically included in Tool Search after their tool\nprovider is loaded. They do not need to be manually assigned to a category.\nSkills use a separate discovery system:\n\n~/.hermes/skills/devops/my-skill/SKILL.md is categorized as devops.\nA skill directly under ~/.hermes/skills/ has no category.\nThe skill name and description come from YAML frontmatter in SKILL.md.\nDirectory additions are noticed automatically. An in-place edit may take up\nto 30 seconds to appear because of the skill cache.\n\nTool Search therefore does not decide skill categories. The first directory\nbelow ~/.hermes/skills/ does.\nRollback\nThe pre-change configuration is preserved at:\n/home/hermes/.hermes/backups/config.yaml.20260725-before-tool-search\nRestore that file only if Tool Search causes a confirmed regression, validate\nthe YAML, and restart hermes-gateway.service.\nRelated\n\nhermes-agent\nskills-index\ntool-calling\ngateway-resource-resilience\n"},"comparisons/index":{"slug":"comparisons/index","filePath":"comparisons/index.md","title":"comparisons","links":[],"tags":[],"content":""},"concepts/index":{"slug":"concepts/index","filePath":"concepts/index.md","title":"concepts","links":[],"tags":[],"content":""},"experiments/active/index":{"slug":"experiments/active/index","filePath":"experiments/active/index.md","title":"active","links":[],"tags":[],"content":""},"incidents/index":{"slug":"incidents/index","filePath":"incidents/index.md","title":"incidents","links":[],"tags":[],"content":""},"infrastructure/index":{"slug":"infrastructure/index","filePath":"infrastructure/index.md","title":"infrastructure","links":[],"tags":[],"content":""},"queries/index":{"slug":"queries/index","filePath":"queries/index.md","title":"queries","links":[],"tags":[],"content":""},"reports/index":{"slug":"reports/index","filePath":"reports/index.md","title":"reports","links":[],"tags":[],"content":""},"runbooks/index":{"slug":"runbooks/index","filePath":"runbooks/index.md","title":"runbooks","links":[],"tags":[],"content":""},"systems/index":{"slug":"systems/index","filePath":"systems/index.md","title":"systems","links":[],"tags":[],"content":""},"tags/hermes":{"slug":"tags/hermes","filePath":"tags/hermes.md","title":"hermes","links":[],"tags":[],"content":""},"tags/chaos":{"slug":"tags/chaos","filePath":"tags/chaos.md","title":"chaos","links":[],"tags":[],"content":""},"tags/ct450":{"slug":"tags/ct450","filePath":"tags/ct450.md","title":"ct450","links":[],"tags":[],"content":""},"tags/experimental":{"slug":"tags/experimental","filePath":"tags/experimental.md","title":"experimental","links":[],"tags":[],"content":""},"tags/agent":{"slug":"tags/agent","filePath":"tags/agent.md","title":"agent","links":[],"tags":[],"content":""},"tags/production":{"slug":"tags/production","filePath":"tags/production.md","title":"production","links":[],"tags":[],"content":""},"tags/ct460":{"slug":"tags/ct460","filePath":"tags/ct460.md","title":"ct460","links":[],"tags":[],"content":""},"tags/agents":{"slug":"tags/agents","filePath":"tags/agents.md","title":"agents","links":[],"tags":[],"content":""},"tags/estate":{"slug":"tags/estate","filePath":"tags/estate.md","title":"estate","links":[],"tags":[],"content":""},"tags/overview":{"slug":"tags/overview","filePath":"tags/overview.md","title":"overview","links":[],"tags":[],"content":""},"tags/nanobot":{"slug":"tags/nanobot","filePath":"tags/nanobot.md","title":"nanobot","links":[],"tags":[],"content":""},"tags/ct333":{"slug":"tags/ct333","filePath":"tags/ct333.md","title":"ct333","links":[],"tags":[],"content":""},"tags/scout":{"slug":"tags/scout","filePath":"tags/scout.md","title":"scout","links":[],"tags":[],"content":""},"tags/openclaw":{"slug":"tags/openclaw","filePath":"tags/openclaw.md","title":"openclaw","links":[],"tags":[],"content":""},"tags/open-claw":{"slug":"tags/open-claw","filePath":"tags/open-claw.md","title":"open-claw","links":[],"tags":[],"content":""},"tags/vm403":{"slug":"tags/vm403","filePath":"tags/vm403.md","title":"vm403","links":[],"tags":[],"content":""},"tags/gateway":{"slug":"tags/gateway","filePath":"tags/gateway.md","title":"gateway","links":[],"tags":[],"content":""},"tags/qwenpaw":{"slug":"tags/qwenpaw","filePath":"tags/qwenpaw.md","title":"qwenpaw","links":[],"tags":[],"content":""},"tags/ct465":{"slug":"tags/ct465","filePath":"tags/ct465.md","title":"ct465","links":[],"tags":[],"content":""},"tags/paseo":{"slug":"tags/paseo","filePath":"tags/paseo.md","title":"paseo","links":[],"tags":[],"content":""},"tags/zeroclaw":{"slug":"tags/zeroclaw","filePath":"tags/zeroclaw.md","title":"zeroclaw","links":[],"tags":[],"content":""},"tags/daemon":{"slug":"tags/daemon","filePath":"tags/daemon.md","title":"daemon","links":[],"tags":[],"content":""},"tags/comparison":{"slug":"tags/comparison","filePath":"tags/comparison.md","title":"comparison","links":[],"tags":[],"content":""},"tags/browser":{"slug":"tags/browser","filePath":"tags/browser.md","title":"browser","links":[],"tags":[],"content":""},"tags/memory":{"slug":"tags/memory","filePath":"tags/memory.md","title":"memory","links":[],"tags":[],"content":""},"tags/routing":{"slug":"tags/routing","filePath":"tags/routing.md","title":"routing","links":[],"tags":[],"content":""},"tags/steel":{"slug":"tags/steel","filePath":"tags/steel.md","title":"steel","links":[],"tags":[],"content":""},"tags/camofox":{"slug":"tags/camofox","filePath":"tags/camofox.md","title":"camofox","links":[],"tags":[],"content":""},"tags/decision":{"slug":"tags/decision","filePath":"tags/decision.md","title":"decision","links":[],"tags":[],"content":""},"tags/concept":{"slug":"tags/concept","filePath":"tags/concept.md","title":"concept","links":[],"tags":[],"content":""},"tags/model":{"slug":"tags/model","filePath":"tags/model.md","title":"model","links":[],"tags":[],"content":""},"tags/inference":{"slug":"tags/inference","filePath":"tags/inference.md","title":"inference","links":[],"tags":[],"content":""},"tags/terminal":{"slug":"tags/terminal","filePath":"tags/terminal.md","title":"terminal","links":[],"tags":[],"content":""},"tags/tool-calling":{"slug":"tags/tool-calling","filePath":"tags/tool-calling.md","title":"tool-calling","links":[],"tags":[],"content":""},"tags/system":{"slug":"tags/system","filePath":"tags/system.md","title":"system","links":[],"tags":[],"content":""},"tags/decisions":{"slug":"tags/decisions","filePath":"tags/decisions.md","title":"decisions","links":[],"tags":[],"content":""},"tags/provider":{"slug":"tags/provider","filePath":"tags/provider.md","title":"provider","links":[],"tags":[],"content":""},"tags/automation":{"slug":"tags/automation","filePath":"tags/automation.md","title":"automation","links":[],"tags":[],"content":""},"tags/shell":{"slug":"tags/shell","filePath":"tags/shell.md","title":"shell","links":[],"tags":[],"content":""},"tags/rollback":{"slug":"tags/rollback","filePath":"tags/rollback.md","title":"rollback","links":[],"tags":[],"content":""},"tags/experiment":{"slug":"tags/experiment","filePath":"tags/experiment.md","title":"experiment","links":[],"tags":[],"content":""},"tags/experiments":{"slug":"tags/experiments","filePath":"tags/experiments.md","title":"experiments","links":[],"tags":[],"content":""},"tags/mnemosyne":{"slug":"tags/mnemosyne","filePath":"tags/mnemosyne.md","title":"mnemosyne","links":[],"tags":[],"content":""},"tags/incident":{"slug":"tags/incident","filePath":"tags/incident.md","title":"incident","links":[],"tags":[],"content":""},"tags/infrastructure":{"slug":"tags/infrastructure","filePath":"tags/infrastructure.md","title":"infrastructure","links":[],"tags":[],"content":""},"tags/backups":{"slug":"tags/backups","filePath":"tags/backups.md","title":"backups","links":[],"tags":[],"content":""},"tags/credentials":{"slug":"tags/credentials","filePath":"tags/credentials.md","title":"credentials","links":[],"tags":[],"content":""},"tags/security":{"slug":"tags/security","filePath":"tags/security.md","title":"security","links":[],"tags":[],"content":""},"tags/docker":{"slug":"tags/docker","filePath":"tags/docker.md","title":"docker","links":[],"tags":[],"content":""},"tags/homelab":{"slug":"tags/homelab","filePath":"tags/homelab.md","title":"homelab","links":[],"tags":[],"content":""},"tags/network":{"slug":"tags/network","filePath":"tags/network.md","title":"network","links":[],"tags":[],"content":""},"tags/cloudflare":{"slug":"tags/cloudflare","filePath":"tags/cloudflare.md","title":"cloudflare","links":[],"tags":[],"content":""},"tags/hosts":{"slug":"tags/hosts","filePath":"tags/hosts.md","title":"hosts","links":[],"tags":[],"content":""},"tags/containers":{"slug":"tags/containers","filePath":"tags/containers.md","title":"containers","links":[],"tags":[],"content":""},"tags/paths":{"slug":"tags/paths","filePath":"tags/paths.md","title":"paths","links":[],"tags":[],"content":""},"tags/proxmox":{"slug":"tags/proxmox","filePath":"tags/proxmox.md","title":"proxmox","links":[],"tags":[],"content":""},"tags/virtualization":{"slug":"tags/virtualization","filePath":"tags/virtualization.md","title":"virtualization","links":[],"tags":[],"content":""},"tags/tailscale":{"slug":"tags/tailscale","filePath":"tags/tailscale.md","title":"tailscale","links":[],"tags":[],"content":""},"tags/vpn":{"slug":"tags/vpn","filePath":"tags/vpn.md","title":"vpn","links":[],"tags":[],"content":""},"tags/networking":{"slug":"tags/networking","filePath":"tags/networking.md","title":"networking","links":[],"tags":[],"content":""},"tags/query":{"slug":"tags/query","filePath":"tags/query.md","title":"query","links":[],"tags":[],"content":""},"tags/research":{"slug":"tags/research","filePath":"tags/research.md","title":"research","links":[],"tags":[],"content":""},"tags/reports":{"slug":"tags/reports","filePath":"tags/reports.md","title":"reports","links":[],"tags":[],"content":""},"tags/uber-eats":{"slug":"tags/uber-eats","filePath":"tags/uber-eats.md","title":"uber-eats","links":[],"tags":[],"content":""},"tags/finance":{"slug":"tags/finance","filePath":"tags/finance.md","title":"finance","links":[],"tags":[],"content":""},"tags/runbook":{"slug":"tags/runbook","filePath":"tags/runbook.md","title":"runbook","links":[],"tags":[],"content":""},"tags/wiki":{"slug":"tags/wiki","filePath":"tags/wiki.md","title":"wiki","links":[],"tags":[],"content":""},"tags/backup":{"slug":"tags/backup","filePath":"tags/backup.md","title":"backup","links":[],"tags":[],"content":""},"tags/cron":{"slug":"tags/cron","filePath":"tags/cron.md","title":"cron","links":[],"tags":[],"content":""},"tags/jobs":{"slug":"tags/jobs","filePath":"tags/jobs.md","title":"jobs","links":[],"tags":[],"content":""},"tags/notion":{"slug":"tags/notion","filePath":"tags/notion.md","title":"notion","links":[],"tags":[],"content":""},"tags/dashboard":{"slug":"tags/dashboard","filePath":"tags/dashboard.md","title":"dashboard","links":[],"tags":[],"content":""},"tags/multiplexer":{"slug":"tags/multiplexer","filePath":"tags/multiplexer.md","title":"multiplexer","links":[],"tags":[],"content":""},"tags/telegram":{"slug":"tags/telegram","filePath":"tags/telegram.md","title":"telegram","links":[],"tags":[],"content":""},"tags/profiles":{"slug":"tags/profiles","filePath":"tags/profiles.md","title":"profiles","links":[],"tags":[],"content":""},"tags/providers":{"slug":"tags/providers","filePath":"tags/providers.md","title":"providers","links":[],"tags":[],"content":""},"tags/health":{"slug":"tags/health","filePath":"tags/health.md","title":"health","links":[],"tags":[],"content":""},"tags/recovery":{"slug":"tags/recovery","filePath":"tags/recovery.md","title":"recovery","links":[],"tags":[],"content":""},"tags/cua-driver":{"slug":"tags/cua-driver","filePath":"tags/cua-driver.md","title":"cua-driver","links":[],"tags":[],"content":""},"tags/restart":{"slug":"tags/restart","filePath":"tags/restart.md","title":"restart","links":[],"tags":[],"content":""},"tags/openrouter":{"slug":"tags/openrouter","filePath":"tags/openrouter.md","title":"openrouter","links":[],"tags":[],"content":""},"tags/fallback":{"slug":"tags/fallback","filePath":"tags/fallback.md","title":"fallback","links":[],"tags":[],"content":""},"tags/update":{"slug":"tags/update","filePath":"tags/update.md","title":"update","links":[],"tags":[],"content":""},"tags/profile":{"slug":"tags/profile","filePath":"tags/profile.md","title":"profile","links":[],"tags":[],"content":""},"tags/llm":{"slug":"tags/llm","filePath":"tags/llm.md","title":"llm","links":[],"tags":[],"content":""},"tags/proxy":{"slug":"tags/proxy","filePath":"tags/proxy.md","title":"proxy","links":[],"tags":[],"content":""},"tags/router":{"slug":"tags/router","filePath":"tags/router.md","title":"router","links":[],"tags":[],"content":""},"tags/guanaco":{"slug":"tags/guanaco","filePath":"tags/guanaco.md","title":"guanaco","links":[],"tags":[],"content":""},"tags/ollama":{"slug":"tags/ollama","filePath":"tags/ollama.md","title":"ollama","links":[],"tags":[],"content":""},"tags/webui":{"slug":"tags/webui","filePath":"tags/webui.md","title":"webui","links":[],"tags":[],"content":""},"tags/desktop":{"slug":"tags/desktop","filePath":"tags/desktop.md","title":"desktop","links":[],"tags":[],"content":""},"tags/homepage":{"slug":"tags/homepage","filePath":"tags/homepage.md","title":"homepage","links":[],"tags":[],"content":""},"tags/monitoring":{"slug":"tags/monitoring","filePath":"tags/monitoring.md","title":"monitoring","links":[],"tags":[],"content":""},"tags/messaging":{"slug":"tags/messaging","filePath":"tags/messaging.md","title":"messaging","links":[],"tags":[],"content":""},"tags/discord":{"slug":"tags/discord","filePath":"tags/discord.md","title":"discord","links":[],"tags":[],"content":""},"tags/email":{"slug":"tags/email","filePath":"tags/email.md","title":"email","links":[],"tags":[],"content":""},"tags/integrations":{"slug":"tags/integrations","filePath":"tags/integrations.md","title":"integrations","links":[],"tags":[],"content":""},"tags/backend":{"slug":"tags/backend","filePath":"tags/backend.md","title":"backend","links":[],"tags":[],"content":""},"tags/obsidian":{"slug":"tags/obsidian","filePath":"tags/obsidian.md","title":"obsidian","links":[],"tags":[],"content":""},"tags/notes":{"slug":"tags/notes","filePath":"tags/notes.md","title":"notes","links":[],"tags":[],"content":""},"tags/vault":{"slug":"tags/vault","filePath":"tags/vault.md","title":"vault","links":[],"tags":[],"content":""},"tags/opencode":{"slug":"tags/opencode","filePath":"tags/opencode.md","title":"opencode","links":[],"tags":[],"content":""},"tags/models":{"slug":"tags/models","filePath":"tags/models.md","title":"models","links":[],"tags":[],"content":""},"tags/subscription":{"slug":"tags/subscription","filePath":"tags/subscription.md","title":"subscription","links":[],"tags":[],"content":""},"tags/limits":{"slug":"tags/limits","filePath":"tags/limits.md","title":"limits","links":[],"tags":[],"content":""},"tags/scheduled":{"slug":"tags/scheduled","filePath":"tags/scheduled.md","title":"scheduled","links":[],"tags":[],"content":""},"tags/skills":{"slug":"tags/skills","filePath":"tags/skills.md","title":"skills","links":[],"tags":[],"content":""},"tags/persistence":{"slug":"tags/persistence","filePath":"tags/persistence.md","title":"persistence","links":[],"tags":[],"content":""},"tags/index":{"slug":"tags/index","filePath":"tags/index.md","title":"Tag Index","links":[],"tags":[],"content":""}}