Tony0410
77250ea8c9
- New incident page documenting CT299 SFTPGo outage (02:18 AWST) - Root cause: swap exhaustion (43Gi, 100%) killed tailscaled; 5 stale lxc-attach PIDs blocked pct start - Resolution: killed PIDs 3037951/3334928/3352203/3402511/4004148, pct stop/start 299 - Updated current-state.md with Recent Changes (2026-09-02), bumped updated to 2026-09-04 - Updated log.md with incident entry - Updated index.md last-updated date No secrets written. Verified end-to-end: tailscale direct, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0.
86 lines
11 KiB
Markdown
86 lines
11 KiB
Markdown
---
|
||
title: Current State
|
||
type: concept
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-09-04
|
||
review_after: 2026-10-22
|
||
confidence: high
|
||
tags: [system]
|
||
sources: []
|
||
---
|
||
|
||
# Current State
|
||
Updated: 2026-09-04
|
||
|
||
## Recent Changes (2026-09-02)
|
||
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
|
||
|
||
## Attested facts
|
||
- **Config version:** 33 — Verified, `~/.hermes/config.yaml`, 2026-07-25
|
||
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
|
||
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
|
||
- **Main provider:** `custom:litellm` → `http://litellm:4000/v1` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
||
- **Default model:** `freellmapi-auto` — Verified, `~/.hermes/config.yaml`, 2026-07-22
|
||
- **Fallback/aux provider:** `custom:omni` → `http://omni:20128/v1` — Verified, config, 2026-08-13 (migrated from `custom:omniroute`, CT600)
|
||
- **OmniRoute host:** `omni.kangaroo-eel.ts.net` → `100.93.204.4` — Verified, DNS resolution, 2026-08-13
|
||
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
|
||
- **Memory enabled:** true — Verified, config, 2026-07-22
|
||
- **Streaming enabled:** true — Verified, config, 2026-07-22
|
||
- **Dashboard public URL:** `https://hermes.martinwa.org/` — Verified, config, 2026-07-22
|
||
- **Pet config:** enabled/slug null — Verified, config, 2026-07-22
|
||
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
|
||
|
||
## Recent Changes (2026-08-02)
|
||
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
|
||
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
|
||
- **Uber Eats report upgraded:** Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with `camofox-browser-automation` + `browser-backend-verification` skills.
|
||
- **GitHub Trending Radar — Daily Digest removed:** Stopped by user on 2026-08-02. Weekly roundup still active.
|
||
- **Ops profile renamed:** Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
|
||
- **Wiki audit cron job created:** Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
|
||
- **Ops profile Mnemosyne enabled:** Mnemosyne plugin enabled on ops profile with `profile_isolation: true` — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: `~/.hermes/profiles/ops/config.yaml` (`memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`).
|
||
- **OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04):** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
|
||
- **OmniRoute three-layer fix (2026-08-07):** npm package `.env` override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See [[systems/omniroute]].
|
||
- **Nanobot status degraded (2026-08-07):** CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See [[systems/nanobot]].
|
||
- **CT460 memory bumped (2026-08-07):** 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.
|
||
|
||
## Recent Changes (2026-08-08)
|
||
- **Cron job model repinning:** 3 jobs repinned from dead `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker repinned from dead `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named cron jobs (e.g. `vertex/gemini-2.5-flash`, `gpt-5.6-terra`, `deepseek-v4-flash-free`) repinned to `smart-route`. Daily Marketing Job Radar swapped to `gpt-5.6-terra`.
|
||
- **Hermes UI vite config revert:** Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build now serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
|
||
- **Phase One estate review completed:** Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, and CT333 Nanobot has a live `jobs.json` with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.
|
||
|
||
## Recent Changes (2026-08-09)
|
||
- **Portainer password reset (CT245):** Admin user is `anthony` (not `admin`). Password was forgotten; reset via `docker run --rm -v portainer_data:/data portainer/helper-reset-password` after stopping the Portainer container. Verified end-to-end with `POST /api/auth` → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on `:9443` (see [[systems/homepage-dashboard]]). Recovery procedure: stop container → run helper image → restart → verify with auth API.
|
||
- **SFTPGo security review (CT299):** Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path `//` resolves to `/` — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is `root`, (4) quota value `F1` on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.
|
||
|
||
## Recent Changes (2026-08-10)
|
||
- **Cron job failures — session storage issue:** Four cron jobs failed on 2026-08-10 with two distinct error patterns:
|
||
- **TimeoutError (idle for 601–602s):** "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
|
||
- **RuntimeError (session storage):** "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
|
||
- **Root cause:** Disk at 83% (11GB free on 65GB). `state.db` is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
|
||
- **Impact:** News digests and GitHub Trending did not deliver. Job Radar did not run.
|
||
- **See:** [[systems/scheduled-tasks]] Known Issues section.
|
||
- **OpenCode WebSocket proxy errors:** Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, `opencode serve`), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.
|
||
|
||
## Recent Changes (2026-08-14)
|
||
- **OmniRoute factory reset on CT601:** Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See [[systems/omniroute]].
|
||
- **Hermes default model changed:** `auto/best-free` → `resilient-free` in `~/.hermes/config.yaml`. `auto/best-free` was misconfigured (no combo definition, routed to paid `gpt-5.6-sol`).
|
||
- **Multiple free combos broken by upstream model ID changes:** OpenRouter Poolside `laguna-s-2-1:free` → `laguna-s-2.1:free` (dash→dot); Groq `llama-4-scout-17b-16e-instruct` retired. Affected: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`.
|
||
- **Guanaco router updated:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See [[systems/guanaco]].
|
||
- **GPT-5.6 Luna research:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See [[systems/model-providers]].
|
||
- **Hermes UI restart:** Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
|
||
- **LiteLLM routing issue identified:** Hermes was previously pointed at LiteLLM (`http://litellm:4000/v1`, model `free-auto`) causing 6–26s per call. OmniRoute `auto/best-free` tested at 265ms. Config now points at OmniRoute.
|
||
|
||
## Recent Changes (2026-08-13)
|
||
- **OmniRoute migrated CT600 → CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Backup: `~/.hermes/config.yaml.bak.omniroute-removal`. See [[systems/omniroute]].
|
||
- **Evening stand-down briefing weather bugfix:** Weather collector returns a plain string (`+16°C Sunny`), but the three dashboard widgets (`evening.html`, `evening-grid.html`, `evening-briefing.html`) read object fields → `undefined`. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.
|
||
|
||
## Related
|
||
- [[systems/browser-backend]]
|
||
- [[systems/mnemosyne]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[comparisons/steel-vs-camoufox]]
|
||
- [[decisions/rollback-steel-to-camofox]]
|