151 lines
31 KiB
Markdown
151 lines
31 KiB
Markdown
---
|
||
title: Current State
|
||
type: concept
|
||
status: active
|
||
created: 2026-07-22
|
||
updated: 2026-10-07
|
||
verified_on: 2026-10-07
|
||
review_after: 2026-10-31
|
||
confidence: high
|
||
tags: [system]
|
||
sources: [live-config, live-cron-list, live-hermes-version]
|
||
---
|
||
|
||
# Current State
|
||
Updated: 2026-10-07
|
||
|
||
## Recent Changes (2026-10-07)
|
||
- **Cron fleet green for a second consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters read 0. Window tally (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00 AWST): **180 `completed`, 0 `failed`, 1 `running`** (the nightly audit itself). **No interactive (non-cron) sessions** in the window — every session was cron. No new incidents raised or resolved in the window.
|
||
- **The stocktake job proved its repin — first fully automatic run since 2026-09-25.** `1a8e90ea2271` (`wiki-stocktake-maintenance`, 03:00 daily) ran on schedule **2026-10-06 03:00:31 → 03:03:32** on its new `custom:omni-101` / `free-stack` pin and completed cleanly. Two of its own incidents (`069d7834bd68`, `7061381d4259`) flipped to `resolved` at 03:03:31. It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt`, rebuilt Quartz and committed to Gitea as `33f4e95`. Its stale pre-repin `last_status: error` is now cleared — live `last_status: ok`, streak 0. **The daily manual `wiki-maintain.sh` run is no longer needed.** See [[systems/scheduled-tasks]].
|
||
- **Three jobs still carry a stale pre-repin `last_status: error`** — `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` (down from four). Each clears on its next scheduled slot (Fri 08:00 / Mon 03:00 / 1st 09:00); none is a live failure. The bare-`telegram` target warning still stands on `066bb5191280` and `0e753e9720fd` (unchanged).
|
||
- **Wiki vault mirror refreshed.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` had drifted to 2026-10-03 / 344 skills while the canonical copy regenerated to 2026-10-06 / 346; this audit re-mirrored it along with the pages below. The two-wiki-tree drift remains a standing open item.
|
||
|
||
## Recent Changes (2026-10-06)
|
||
- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||
- **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||
- **GitHub Trending Radar's revoked-token diagnosis is refuted.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
|
||
- **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
|
||
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session.
|
||
|
||
## Recent Changes (2026-10-05)
|
||
- **Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7.** Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is `79f9bb41ecf5` (Weekly Review, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `cliprox/terra` error the incident predicted: `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That is the last untested job on the pin, so the provider-connection auth expiry is now **3 of 3 confirmed** — settled as a credential fault, not per-job. No review was produced; `delivery_outcome: delivered` again because the failure notice is the deliverable. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||
- **`1a8e90ea2271` failed again — 2026-10-04 03:00 (HTTP 410).** The `wiki-stocktake-maintenance` job remains pinned to the Ollama-retired `qwen3.5:397b`; no change, still needs a repin. Its script is healthy. See [[systems/scheduled-tasks]].
|
||
- **No interactive (non-cron) sessions in the window** — all 10 sessions active in the past 24h were cron. The only durable changes are cron-fleet status, reflected above.
|
||
- **Streak-number caveat for future audits:** the `N failures in a row` figure in `hermes cron list` is the `failure_streak` counter from `jobs.json`, not a DB-derived count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows), so DB recounts undercount older streaks. Live counters 2026-10-05: `1a8e90ea2271` ×11, `dca8482e4f76` ×7, `fad5a86979d4` ×3. Quote the live counter, and note the retention floor.
|
||
|
||
## Recent Changes (2026-10-04)
|
||
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
|
||
- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]].
|
||
- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
|
||
- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]].
|
||
- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
|
||
- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line.
|
||
|
||
## Recent Changes (2026-10-03)
|
||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||
- **Streaks recounted from `executions.db` — one number corrected:** `dca8482e4f76` 6 → **7** (one more daily run), but `1a8e90ea2271` **8 → 7** — the ×8 carried since 2026-10-01 was an overcount; that job has exactly 7 recorded runs and all 7 failed. Recount before quoting a streak.
|
||
- **A delivered notice is not a successful job.** Both `cliprox/terra` failures are recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable — the execution `status` is `failed` and no digest/plan was produced. Judge cron health on the `status` column, not the delivery outcome.
|
||
- **Resolved — `1b23c042e17d` stale `delivery_failed` flag cleared:** the 2026-09-30 retarget to `telegram:-1004321904721:102` worked exactly as predicted; its 2026-10-02 run delivered (then failed on the model pin). The prediction held.
|
||
- **[[systems/skills-index]] regenerated by hand: 341 → 344 skills.** The daily stocktake job (`1a8e90ea2271`) is still blocked on a retired model (HTTP 410, `qwen3.5:397b`), but its **script is healthy** — `wiki-maintain.sh` was run manually on 2026-10-03 and completed cleanly, refreshing the index and writing `raw/stocktake/freshness-20261003.txt`. One new skill appeared: `fleet-wide-failure-triage` (devops). Only the scheduled wrapper is broken; every non-model step of the stocktake now needs a manual run each day until it is repinned.
|
||
- **Work mailbox blocker documented — and it is older than it looks.** A new incident page records that the `work` Outlook account has been unreachable since **2026-08-26**, not this week: the account uses password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. The TCP/TLS path is fine — it is a tenant policy wall, not a bad password, and re-running the credential helper can never fix it. It needs an OAuth2 path. No cron job polls that mailbox, so this is an on-demand capability gap. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||
- **GitHub Trending Radar token claim downgraded to "unverified":** the revoked-bot-token (`getMe` → 401) diagnosis could not be reproduced — `c0a9fa314542`'s last recorded run (2026-09-30 19:22, off-schedule) **completed and delivered**. Re-confirm at the next scheduled run (Mon 2026-10-05 09:00) before acting on it.
|
||
|
||
## Recent Changes (2026-10-02)
|
||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **3 → 6**: two *new* failures landed on 2026-10-01 —
|
||
- `06a0a1c8704f` (AI Newsletter Curation for Patrick, Thu 10:00) — `HTTP 401 … All 1 connection(s) authentication expired — please reconnect in the dashboard` on provider `custom:Omniroute:20128`, model `cliprox/terra`. Its failure notice reached topic 346; no digest was produced. **Not self-recovering** — needs the OmniRoute connection re-authenticated. Two other enabled jobs share that pin (`1b23c042e17d`, `79f9bb41ecf5`) and will fail next run.
|
||
- `0e753e9720fd` (Facebook Birthday Monthly Scrape, 09:00) — `HTTP 400`, `nvidia/deepseek-ai/deepseek-v4-flash` absent from the active NVIDIA catalog, confirming the earlier diagnosis that this is a **shared-pin** problem (sibling `fad5a86979d4` uses the same model).
|
||
- `066bb5191280` (Uber Eats Monthly Spend Report, 02:11) — report generated correctly; **delivery** failed with `configured thread_id 34795 for telegram:1793951355 was not found`.
|
||
- **Carried-over failures, streaks advanced:** `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` → streak 6; `1a8e90ea2271` HTTP 410 (retired `qwen3.5:397b`) → streak 8; `fad5a86979d4` HTTP 400 → streak 3. All three still need a model repin.
|
||
- **Telegram delivery semantics corrected — a "failed" delivery is not always a lost message.** Against `~/.hermes/cron/executions.db`, the 2026-10-01 failures carry the *fallback* variant: `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id`. Content reaches the DM, yet the run is recorded `delivery_outcome: failed`. The 2026-09-30 incident's `Message thread not found` variant is the one that truly loses the message. Both variants are now documented in [[systems/messaging-integrations]].
|
||
- **The wiki has two trees, and they have drifted in both directions (new finding, 2026-10-02):** the canonical source is **`/home/hermes/workspace/agent-estate-wiki`** — it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build (`serve.py`, `package.json`, `public/`) and is what `wiki-maintain.sh` commits. **`/home/hermes/wiki/agent-estate-wiki`** is the *Obsidian/Ignis vault copy* (pages nested under `wiki/`, no git remote) that the iPad reads. The nightly audit is told to write to the vault path, so edits that skip the canonical tree never reach Gitea or the published site. Worse, the trees have diverged backwards too: `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist **only in the vault**, while `raw/stocktake/*`, `incidents/2026-09-02-sftpgo-outage.md`, `reports/communities-wa-social-media-audit-2026.md` and `agents/` exist **only in canonical**. `index.md` in canonical links `[[systems/ignis]]` and `[[systems/nextcloud]]`, both of which are broken there. Any one-way sync will silently delete pages — reconcile before trusting either tree. See [[systems/scheduled-tasks]].
|
||
- **Skills inventory:** 344 `SKILL.md` under `~/.hermes/skills/` (live count 2026-10-02), against 341 in the last generated [[systems/skills-index]] (2026-09-27). The page cannot refresh while the stocktake job stays broken.
|
||
|
||
## Recent Changes (2026-10-01)
|
||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **4 → 3**: `cbfe434a3b5e` returned to `ok` as predicted on 2026-09-30. The residual `1a8e90ea2271` dispatch error also cleared — its current failure is a different, upstream problem (below).
|
||
- **Telegram delivery failures root-caused and fixed:** the target, not the topic, was wrong — a bare `deliver: "telegram"` resolves to the **home DM (`1793951355`)**, where group topic IDs cannot exist, so sends hard-failed and the message was lost. `1b23c042e17d` and `c8d52904e1d6` retargeted to `telegram:-1004321904721:102` and verified. Three jobs still carry bare `telegram` targets (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). See [[systems/messaging-integrations]].
|
||
- **Three jobs pinned to models that no longer exist upstream — cannot self-recover:** `1a8e90ea2271` → `qwen3.5:397b` retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`, streak 5); `fad5a86979d4` → `nvidia/…/deepseek-v4-flash` absent from the live catalog (HTTP 400, streak 3). Repin to the `smart-route` alias. See [[systems/model-providers]].
|
||
- **GitHub Trending Radar — revoked Telegram bot token:** `c0a9fa314542` builds its digest then fails the push (`getMe` → 401). Token in `/home/hermes/workspace/github-trending-radar/config.json` needs replacing from @BotFather.
|
||
- **Memory hygiene pass (2026-09-30):** 36 gateway background-process notifications had been written into Mnemosyne as `[USER] [Anthony Martin]` memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts `hermes_cron_model_default` and the job-radar model corrected. See [[systems/mnemosyne]].
|
||
- **Provider stack rebuilt from live config:** 5 custom providers (`omni-101`, `Omniroute:20128`, `litellm`, `command-code`, `guacano`). The provider key `custom:omni` **no longer exists**; two cron jobs (`e530c89f2be0`, `d40085da631c`) still name it but have not failed, because the default `custom` block resolves to the same base URL.
|
||
- **Vision routing — request recorded, not applied:** Anthony asked on 2026-09-30 that vision calls use the local Ollama at `http://anthonys-mac-mini:11434` (gemma4). Live config still points at `nous / stepfun/step-3.7-flash:free` as at 2026-10-01.
|
||
|
||
## Recent Changes (2026-09-30)
|
||
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
|
||
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
|
||
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
|
||
- **Open — Telegram thread delivery failures (root cause found 2026-10-01):** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Cause was the delivery target, not the topic — a bare `telegram` resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
|
||
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
|
||
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
|
||
|
||
## Recent Changes (2026-09-02)
|
||
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
|
||
|
||
## Attested facts
|
||
- **Config version:** `_config_version: 46` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously recorded as 33 on 2026-07-25)
|
||
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
|
||
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
|
||
- **Main provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `custom:litellm` → `litellm:4000`, then `custom:omni` → `omni.kangaroo-eel.ts.net`)
|
||
- **Default model:** `smart-route` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `freellmapi-auto`)
|
||
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` — Verified, config, 2026-09-30
|
||
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` — Verified, config, 2026-09-30
|
||
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
|
||
- **Memory enabled:** true — Verified, config, 2026-07-22
|
||
- **Streaming enabled:** true — Verified, config, 2026-07-22
|
||
- **Dashboard public URL:** `https://hermes.martinwa.org/` — Verified, config, 2026-07-22
|
||
- **Pet config:** enabled/slug null — Verified, config, 2026-07-22
|
||
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
|
||
|
||
## Recent Changes (2026-08-02)
|
||
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
|
||
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
|
||
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
|
||
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
|
||
- **Uber Eats report upgraded:** Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with `camofox-browser-automation` + `browser-backend-verification` skills.
|
||
- **GitHub Trending Radar — Daily Digest removed:** Stopped by user on 2026-08-02. Weekly roundup still active.
|
||
- **Ops profile renamed:** Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
|
||
- **Wiki audit cron job created:** Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
|
||
- **Ops profile Mnemosyne enabled:** Mnemosyne plugin enabled on ops profile with `profile_isolation: true` — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: `~/.hermes/profiles/ops/config.yaml` (`memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`).
|
||
- **OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04):** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
|
||
- **OmniRoute three-layer fix (2026-08-07):** npm package `.env` override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See [[systems/omniroute]].
|
||
- **Nanobot status degraded (2026-08-07):** CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See [[systems/nanobot]].
|
||
- **CT460 memory bumped (2026-08-07):** 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.
|
||
|
||
## Recent Changes (2026-08-08)
|
||
- **Cron job model repinning:** 3 jobs repinned from dead `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker repinned from dead `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named cron jobs (e.g. `vertex/gemini-2.5-flash`, `gpt-5.6-terra`, `deepseek-v4-flash-free`) repinned to `smart-route`. Daily Marketing Job Radar swapped to `gpt-5.6-terra`.
|
||
- **Hermes UI vite config revert:** Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build now serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
|
||
- **Phase One estate review completed:** Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, and CT333 Nanobot has a live `jobs.json` with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.
|
||
|
||
## Recent Changes (2026-08-09)
|
||
- **Portainer password reset (CT245):** Admin user is `anthony` (not `admin`). Password was forgotten; reset via `docker run --rm -v portainer_data:/data portainer/helper-reset-password` after stopping the Portainer container. Verified end-to-end with `POST /api/auth` → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on `:9443` (see [[systems/homepage-dashboard]]). Recovery procedure: stop container → run helper image → restart → verify with auth API.
|
||
- **SFTPGo security review (CT299):** Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path `//` resolves to `/` — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is `root`, (4) quota value `F1` on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.
|
||
|
||
## Recent Changes (2026-08-10)
|
||
- **Cron job failures — session storage issue:** Four cron jobs failed on 2026-08-10 with two distinct error patterns:
|
||
- **TimeoutError (idle for 601–602s):** "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
|
||
- **RuntimeError (session storage):** "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
|
||
- **Root cause:** Disk at 83% (11GB free on 65GB). `state.db` is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
|
||
- **Impact:** News digests and GitHub Trending did not deliver. Job Radar did not run.
|
||
- **See:** [[systems/scheduled-tasks]] Known Issues section.
|
||
- **OpenCode WebSocket proxy errors:** Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, `opencode serve`), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.
|
||
|
||
## Recent Changes (2026-08-14)
|
||
- **OmniRoute factory reset on CT601:** Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See [[systems/omniroute]].
|
||
- **Hermes default model changed:** `auto/best-free` → `resilient-free` in `~/.hermes/config.yaml`. `auto/best-free` was misconfigured (no combo definition, routed to paid `gpt-5.6-sol`).
|
||
- **Multiple free combos broken by upstream model ID changes:** OpenRouter Poolside `laguna-s-2-1:free` → `laguna-s-2.1:free` (dash→dot); Groq `llama-4-scout-17b-16e-instruct` retired. Affected: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`.
|
||
- **Guanaco router updated:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See [[systems/guanaco]].
|
||
- **GPT-5.6 Luna research:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See [[systems/model-providers]].
|
||
- **Hermes UI restart:** Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
|
||
- **LiteLLM routing issue identified:** Hermes was previously pointed at LiteLLM (`http://litellm:4000/v1`, model `free-auto`) causing 6–26s per call. OmniRoute `auto/best-free` tested at 265ms. Config now points at OmniRoute.
|
||
|
||
## Recent Changes (2026-08-13)
|
||
- **OmniRoute migrated CT600 → CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Backup: `~/.hermes/config.yaml.bak.omniroute-removal`. See [[systems/omniroute]].
|
||
- **Evening stand-down briefing weather bugfix:** Weather collector returns a plain string (`+16°C Sunny`), but the three dashboard widgets (`evening.html`, `evening-grid.html`, `evening-briefing.html`) read object fields → `undefined`. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.
|
||
|
||
## Related
|
||
- [[systems/browser-backend]]
|
||
- [[systems/mnemosyne]]
|
||
- [[systems/scheduled-tasks]]
|
||
- [[comparisons/steel-vs-camoufox]]
|
||
- [[decisions/rollback-steel-to-camofox]]
|