Files
agent-estate-wiki/current-state.md
T
Tony0410 987eba72e6 Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
2026-10-30 05:41:11 +08:00

721 lines
101 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Current State
type: concept
status: active
created: 2026-07-22
updated: 2026-10-30
verified_on: 2026-10-30
review_after: 2026-11-07
confidence: high
tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe]
---
# Current State
Updated: 2026-10-30
## Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)
- **Conduit voice calls were repaired — after three faults in series — and the first two real calls in the
product's history are now on record.** Anthony asked why Conduit never answered when he called. In order:
(A) `voice.speech_server.url` held the truncated `wss://…:8443/v1/`, which the plugin dials **verbatim**, so
the Mac answered **403** and the plugin blamed the key; (B) `dashboard.basic_auth.session_ttl_seconds` was the
**12-hour default**, and an expired phone session gets **WS 4401** with no Hermes turn and no error visible on
the phone (the iPhone re-authenticated at **08:00:59**, minutes *after* a silent call); (C) **both** host
providers were dead — `tts.provider: openai` → Mac `/v1/speech` **502**, `stt.provider: groq` → **403 /
Cloudflare 1010**. Fixed 08:02–08:4x: URL → `…:8443/v1/realtime`, TTL → **2 592 000 s (30 d)**, `stt.provider:
local` (faster-whisper `base`), `tts.provider: ollama_local` (6.94 s of real speech verified); dashboard
restarted **08:03:26**. Verified end to end by driving the whole call path, then by **calls at 08:38:04 and
08:49:36** — `conduit-calls.json → history` is no longer empty.
→ [[incidents/2026-10-29-conduit-voice-silence]], [[systems/conduit]], [[systems/hermes-agent]]
- **The Job Radar's two-stage split worked end to end for the first time.** Stage 1 (`dca8482e4f76`,
`deliver: local`) ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`**, and wrote
`~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**,
sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok`. Stage 2 (`5fc292ff0f3b`) consumed the
handoff at **10:40:59**, closed **10:44:48 `completed`** and delivered to topic 1030 in the same second:
**20 pages / 20 `page_ids` / 2 cover letters / `verified_cover_letter_count 2` /
`missing_cover_letter_count 0`**, Notion board **455 → 475**. Both of stage 1's incident rows
(`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**; the 11:30 and 14:30 slots
then no-oped as designed (10 m 46 s / 1 m 51 s).
→ [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]],
[[runbooks/job-radar]]
- **Cron: 2026-10-29 settled 177 `completed` / 0 `failed`** — the first zero-failure day since 2026-10-24 — with
**10 deliveries, all 10 `delivered`, 0 failures**. Non-clean `last_status` **5 → 3**; unclosed incident rows
**4 → 1** (the two radar rows closed `resolved` at 10:16:12). Fleet unchanged at **29 jobs (26 enabled / 3
disabled)** at that audit.
- **Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`:
Gateway connection failed; no local fallback"*), opening incident `ea79a2_d93c3acdc4f2` (`detected`), and the
scheduler **auto-paused the job**. Fleet is now **29 jobs (25 enabled / 4 disabled)**; non-clean
`last_status` is **4** (2 `error` + 2 `delivery_failed`); `ea79a2107e36` carries `failure_streak: 1`.
⚠ **The failure notice itself failed to send** — `hermes cron list` reports `send_path_degraded; live adapter
delivery to telegram:-1004321904721:102 failed`, and `deliveries.db` holds **zero rows for 2026-10-30**, so
the only trace is the incident row. **Unclosed incident rows 1 → 2.** Every other previously-red job
(`dca8482e4f76`, `1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`,
`c0a9fa314542`) has **cleared to `ok` with streak 0** on its own slot. Pins: **18 of the 25 enabled jobs pin
`custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); 7 enabled rows are `no_agent`
jobs with no pin. → [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)
- **The radar's silent budget overrun became a recorded failure — and the radar is now two jobs.** Window
2026-10-27 settled **172 `completed` / 1 `failed`**. The failure is `dca8482e4f76` (Daily Marketing Job
Radar - Discovery): **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed **`failed`**, delivery `delivered`,
and the delivered report opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* —
discovery, verification, dedup and classification completed; **Notion ingestion, cover-letter generation and
the final cover-letter audit did not execute**. The prior day (10-26) had the same overrun but closed
`completed`, hiding it in a green row. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted`
11:03:59, **still open** at this audit (`failure_type: auth` — misclassified). → new page
[[incidents/2026-10-27-job-radar-budget-exhausted]]
- **Response: the radar was split into a two-stage pipeline the same day.** Stage 1 `dca8482e4f76` — Discovery
(`0 7 * * 1-5`, `deliver: local`, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2
`5fc292ff0f3b` — Daily Marketing Job Radar - Write and Letters (`30 11,14 * * 1-5`, Telegram topic 1030,
created **2026-10-27T17:34:15**) lands the Notion entries, cover letters and audit. Two test fires of stage 2
on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both `completed` with delivery `suppressed`; the first
real 07:00 → 11:30 cycle had not yet run. **Fleet 28 → 29 jobs (26 / 3).** → [[runbooks/job-radar]]
- **The 2026-10-26 orphan class did not recur, and its two incident rows closed.** `ea79a2_a4661554a3c2` and
`ea79a2_c3ab2d784ba3` **auto-closed 00:38:32** (unclosed rows **3 → 2**, the 2 being the Facebook birthday
monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog
and every other job were clean all day.
- **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error`
(streak 1) and `08eb86f02d37` re-joined as `delivery_queued`. Still non-clean: `0e753e9720fd` (error),
`fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.**
- **Also updated:** `incidents/2026-10-27-job-radar-budget-exhausted.md` (new),
`incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (recurrence check), `runbooks/job-radar.md`,
`systems/scheduled-tasks.md`, `systems/hermes-agent.md` (TTS provider correction),
`systems/conduit.md` (voice pipeline), `infrastructure/tailscale.md` (peer table re-verified),
`systems/skills-index.md` (regenerated 10-27, committed), `index.md`, `log.md`.
**Nothing removed or archived.** No secrets written.
## Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)
### The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap
`dca8482e4f76` (stage 1, Discovery) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**) and closed **`failed`** on
*"hit the agent iteration cap before the handoff could be written"* — **`failure_streak: 2`**. Stage 1 never
produced a handoff (`~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`), so the *first* real two-stage
cycle delivered **no Notion writes and no cover letters** — the same outcome as 10-27, by a different route.
Both stage-2 slots then did exactly what they were built to do: `5fc292ff0f3b` fired **11:30:27 → 11:31:31** and
**14:30:27 → 14:31:43**, both `completed` with **`delivery_outcome: suppressed`** — *a stage-2 run with no
handoff is a silent no-op, not a failure* (the recovery rule in [[runbooks/job-radar]]), now proven in
production. The two post-split failures also stopped at different ceilings: 10-27 at the **tool-call budget**
(4 h 2 m, writes not started), 10-28 at the **iteration cap** (3 h 32 m, handoff not written) — less of the
pipeline completed despite the shorter wall clock. Open radar incident rows: `dca848_e93a538ade6b` (10-27) and
`dca848_0a2f03b5952b` (10-28), both open. Detail:
[[incidents/2026-10-28-job-radar-stage1-iteration-cap]].
### The 03:00 stocktake failed on a provider 400 — and its incident row never alerted
`1a8e90ea2271` ran **03:00:28 → 03:37:58** (37 m 30 s) and closed **`failed`** (streak 1, delivery
`suppressed`) with `RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate
field 'reasoning_content' at line 1 column 512203` against `custom:omni-101 / free-stack` — the *response* the
provider was asked to deserialize carried a duplicated `reasoning_content` field. It performed its rebuild
(302 skills → `systems/skills-index.md` mtime 03:01:40; freshness report `raw/stocktake/freshness-20261028.txt`
→ **2 stale / 103 fresh / 4 no-date of 109 pages**) and then died **before the commit**, leaving the wiki tree
dirty for the rest of the day. Its incident row `1a8e90_f57ab4cafab8` sits in state **`detected`** with **no
alert** — the only outward sign was the job's own `error` status. The 2026-10-29 audit committed the two
artefacts.
### Steel left the stack: fully removed — Camofox is the sole backend
`browser.cloud_provider` and `browser.camera_provider` are back to **`camofox`** (`engine: auto`, `cdp_url: ""`)
in `config.yaml` (mtime **2026-10-29 00:15:13**), with backups taken at every step
(`config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z` 22:00:57, and
`config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24). The **`browser-steel` plugin was
removed** from `~/.hermes/plugins/`, and no `browser-steel` key remains in `plugins.disabled` or
`plugins.entries`, so the `plugin_steel` toolset is no longer selectable. **Steel is now fully REMOVED, not
merely down:** on **CT201** the containers `steel-browser-api-1` / `steel-browser-ui-1`, the `steel-network`
Docker network, the images (`steel-browser-api:fixed` and both ghcr.io Steel images) and the project dir
`/root/steel-browser` were deleted (~6 GB reclaimed; disk 70G used → 64G used), and the Tailscale Serve routes
`/steel` and `/steel-api` were removed — CT201's remaining routes are only `/` → `127.0.0.1:8085` and `/mgr` →
`127.0.0.1:8084` on **both** `:443` and `:8443`, with nothing listening on 3000, 5173 or 9223. **CT333's second,
separate Steel copy** (`steel-browser`, `ghcr.io/steel-dev/steel-browser:latest`, 5.53 GB, created 2026-05-30)
was removed with its image. Hermes core contains **zero** Steel references, so there is **no Steel fallback path
in the tree** — returning to Steel means redeploying it (the compose file and Dockerfile are archived on CT201
under `/root/_archived-steel/`; the images remain re-pullable from ghcr.io). ⚠ `STEEL_BASE_URL` is no longer in
`~/.hermes/.env` — it survived only in the inherited environment of processes already running at the cleanup,
and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it (a couple of older, unrelated services
still carry it) — **no core Hermes code reads it**, so it is a runtime-only leftover, **not** a config value.
**Camofox is the sole
backend** and healthy: REST `http://100.118.5.51:9377` (and `http://localhost:9377`), `/health` → **200**,
`/tabs` → `{"running":true,"tabs":[]}`. Detail: [[systems/browser-backend]],
[[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]].
> ⚠ **Superseded:** an earlier 2026-10-29 00:3x probe described Steel only as "not reachable — endpoint dark,
> still running on CT201". That was already out of date — Steel is **removed**, not down.
### Fleet and tallies
29 jobs (**26 enabled / 3 disabled**) — unchanged. Non-clean `last_status`: **5** (`0e753e9720fd` error,
`fad5a86979d4` delivery_failed, `dca8482e4f76` error **streak 2**, `066bb5191280` delivery_failed,
`1a8e90ea2271` error **streak 1**). Pins: **12 `smart-route` / 6 `free-stack` / 7 unpinned /
1 `deepseek-paid-smart`**. **2026-10-28 settled 168 completed / 2 failed / 1 running**; **2026-10-29** so far
6 completed / 1 running; **2026-10-27 settled 173 / 1** (revised up by one late run after the 2026-10-28
audit). No Hindsight service is running on this box (`:8888` closed, no `hindsight` units) — the standing
memory-backend correction still holds and needs no change.
## Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)
- **A *planned* gateway restart still orphaned a live cron run — and this time it alerted loudly.** A planned
`SIGTERM` at **2026-10-26 14:33:35** produced a shutdown drain that reported **`timed_out=True` after
just 1.35 s** of its 30 s budget and interrupted `ea79a2107e36` (Mnemosyne Health Monitor, 12:01 slot)
**2 h 34 m** in. It closed **`failed`** on a **new** string — `lost its durable fire claim ownership` —
against the 2026-10-20 orphan's silent `unknown`, and raised **two** `cron_incidents` rows (14:33:37
*unverified / reconcile admission*, 14:35:59 *fire claim*) that are still `alerted`; the failure **was
delivered** at 14:35:59. Two Mnemosyne Watchdog reds followed and **self-resolved at 15:16:12**. Window
2026-10-26 settled **169 / 3** and **all three failures are this one fault** — so the fleet's failure
count is one interruption, not a systemic turn. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
**non-clean `last_status` 5 → 4**; **unclosed incident rows 2 → 3**; **pins unchanged**. →
new page [[incidents/2026-10-26-planned-restart-orphaned-cron-run]]
- **One incident row closed, and the 2026-10-16 503 burst is fully shut.** `c0a9fa_764558b5ae83`
(GitHub Trending Radar) **auto-closed 09:13:00** when its 09:00 slot ran clean — the last open row of the
resource-pressure fault → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. No execution row
carries the 503 string on 10-25 or 10-26. The job stays **`monitoring`**: single-node exposure unchanged.
- **Two quiet degradations worth naming.** `fad5a86979d4`'s 03:08 slot **completed but delivered
`delivery_failed`** — *Thread 34795 not found* (the gateway's own shutdown notice hit the same dead
thread at 14:33:37); and the **radar's 07:00 slot closed `completed` at 09:10 while delivering
“RUN INCOMPLETE — tool-call budget exhausted”** — no Notion writes, no cover letters, batch finished by
hand. That morning the cover-letter PDF pipeline moved to **Chromium over the canonical HTML** with model
**`deepseek-paid-smart`** and a **600 s** dashboard timeout → [[runbooks/job-radar]]
- **Also updated:** `incidents/2026-10-26-planned-restart-orphaned-cron-run.md` (new),
`incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md` (recurrence), `incidents/2026-10-16-omniroute-503-resource-pressure.md`,
`runbooks/job-radar.md`, `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) *(superseded by the 2026-10-27 block above)*
- **A third consecutive clean cron day — and an off-cron fault that is not the fleet's.** Window
2026-10-24: **167 `completed` / 0 `failed`** (+ this audit `running`); **2026-10-23 settled at
170 / 0** (169 at the 10-24 audit). Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**,
**4 non-clean `last_status`**, **2 unclosed incident rows** — nothing opened, nothing closed
in-window — pins unchanged. **14 sessions: 11 `cron` + 3 `desktop`.** →
[[systems/scheduled-tasks]]
- **⚠ New: the agent refused to phone Anthony, three times.** On 2026-10-24 12:53–12:56 he asked to
be called to test Conduit calls (`Call me now to test calls.` → `Yes. You do` → `CAL ME`). All
three replies asserted the agent has **no calling capability** — with **zero tool calls** in the
session — while `conduit_call_user` was one `tool_search` away and his call settings were
permissive. **Capability asserted instead of discovered**; the tool itself is fine. → new page
[[incidents/2026-10-24-conduit-call-request-refused]]
- **Conduit, corrected and updated.** Plugin **`0.69.0` / HEAD `f000bf0` (22 commits on)** while the
gateway process still runs the 10-23 code and the memory-timeout patch is **uncommitted and now
five releases behind HEAD**. Device phone-status contact last 10-23 19:10:30; health ingestion to
10-24 09:59:59; store **569.9 MB / 1,520,229 events / 3,028 `buckets` / 124 `windows`** (the
10-24 page's 308 / 62 was a narrower grouping). ⚠ **`conduit-calls.json → history` is empty** — the
10-22 memory-fix incident's “two calls closed cleanly 00:03:23 / 00:06:57” was a **false positive**
(cron turn-ends, not calls), so that fix still has no positive call evidence. →
[[systems/conduit]], [[incidents/2026-10-22-conduit-memory-start-timeout]]
- **Also updated:** `systems/conduit.md`, `incidents/2026-10-22-conduit-memory-start-timeout.md`
(correction), `systems/scheduled-tasks.md`, `index.md`, `log.md`. **Nothing removed or archived.**
## Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) *(superseded by the 2026-10-25 block above)*
- **A clean cron day, and the 2026-10-09 timeout storm is finally closed.** Window 2026-10-23:
**169 `completed` / 0 `failed`** (+ this audit `running`) — the third zero-failure day in four;
**2026-10-22 settled at 173 / 0**. `1b23c042e17d` (Enhanced Weekend Adventure Planner, weekly Fri 08:00)
ran clean on its 10-23 slot, closing the storm's **last two incident rows**
(`1b23c0_c35703f637b8`, `1b23c0_103cbe04a79b`) at **08:04** — all 12 rows closed, job `ok` again.
**Unclosed incident rows 4 → 2**, **non-clean `last_status` 5 → 4**, **fleet unchanged at 28 jobs
(25 enabled / 3 disabled)**, pins unchanged. → [[incidents/2026-10-09-cron-provider-timeout-storm]],
[[systems/scheduled-tasks]]
- **New: Conduit is documented.** The dashboard plugin + iOS app that puts Hermes on Anthony's phone
(voice calls, cards, push) now has a system page — plugin tree, version `0.64.0` (HEAD `eaa14be`),
upstream `kaishi00/hermes-conduit-notifier`, gateway on `:9119`, one registered device (iPhone, app
`2.0.0`), its state/DB files and their sizes, and the uncommitted patch it carries.
→ [[systems/conduit]]
- **⚠ New fault, found and fixed in-window: Conduit voice calls were being abandoned by the memory
layer.** The plugin hard-coded a **10 s** budget for one request *including the Mnemosyne provider
start*, while Mnemosyne's cold start for a `conduit-voice` session runs **10–45 s**. Three calls hit it
(10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: `Memory provider 'mnemosyne' start for Conduit timed
out; abandoning the call`) and ran **without recall**. Patched in the plugin's working tree to a **60 s**
default (`CONDUIT_MEMORY_TIMEOUT_S` override) plus a `TokenError` guard that proceeds without recall
instead of aborting; the dashboard restarted **18:54:43** and **no abort line exists after it**, with two
calls closing cleanly at 00:03:23 and 00:06:57 on 10-24. **The patch is uncommitted** — a plugin update
reverts it. → [[incidents/2026-10-22-conduit-memory-start-timeout]], [[systems/mnemosyne]]
- **Sessions: 14 in the window** — 13 `cron` + 1 `desktop` (10-23 18:26:12, the phone-call session above);
no other interactive sessions.
- **Stocktake:** the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed
(`250c735`: freshness 2 stale / 94 fresh, skills-index 306). → [[systems/skills-index]]
- **Housekeeping:** frontmatter added to `incidents/2026-07-29-mnemosyne-relay.md` and
`incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md`, both of which sat outside the SCHEMA
(rule 4). Freshness: 102 pages; **3** carry a `verified_on` older than 21 days — `infrastructure/hosts.md`,
`runbooks/provider-health.md`, `runbooks/update-hermes-safely.md` (all 94 d). Nothing removed or archived.
## Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST) *(superseded by the 2026-10-24 entry above)*
- **First zero-failure cron day in three days, and five incidents cleared at once.** Window 2026-10-22: **172
`completed` / 0 `failed`** (+ this audit `running`); 4 runs had already completed on 10-23 by 00:0x. **Five
incident rows closed and none opened** — `0e5198_be2291809e56` + `0e5198_ec2fac98a91d` (08:01:40),
`06a0a1_9e0ff4eedc49` (10:14:34), `c8d529_e8c70250f9be` (18:17:59), `dca848_014affeb6954` (07:59:22) — so
**unclosed rows fall 9 → 4** (3 `alerted` / 1 `detected`). **Non-clean `last_status` falls 9 → 5**:
`1b23c042e17d` (`error`, streak 2, next slot **Fri 2026-10-23 08:00**), `c0a9fa314542` (`error`, 1, Mon 10-26),
`0e753e9720fd` (`error`, 0, 11-01), `fad5a86979d4` + `066bb5191280` (`delivery_failed`, Mon 10-26 / 11-01). All
four cleared job rows had been red since **2026-10-15** (the `omni-101` 503 burst) and each ran clean on its
first slot of 10-22. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **Correction to the 2026-10-21 tally.** A DB recount by both `started_at` and `claimed_at` (AWST) puts
**2026-10-21 at 168 `completed` / 1 `failed`**, three below the 171 / 1 recorded on 10-22. Retention does not
explain it (floor 2026-10-08); the delta is recorded rather than overwritten. → [[systems/scheduled-tasks]]
- **⚠ The stocktake ran six times long, reported `completed`, and committed nothing.** `1a8e90ea2271` ran **2026-10-22
03:00:28 → 04:38:19 (97 m 58 s)** against 16 m 01 s on 10-21, delivery `suppressed`; its regenerated
`systems/skills-index.md` (mtime 03:00) was still uncommitted at audit time and its diagnosed fixes were never
applied. **A `completed` cron row is not evidence the work happened.** → [[incidents/2026-08-15-skills-index-drift]]
- **Skills index fixed at the root tonight.** The page counted **41 parked skills under
`~/.hermes/skills/.archive/`**, and two scripts generated it with different shapes (repo `wiki-maintain.sh`,
`find`-based → 347 with a `## .Archive` section; stale `~/.hermes/scripts/wiki-maintain.sh`, `rglob`-based → 348
without). The repo script now excludes `*/.archive/*`, prints the archived count as an exclusion, and names
`hermes skills list` as the authoritative loadable figure. **Page regenerated: 306 skills across 69 categories**
(was 347). Live counts: filesystem **348**, with symlinks **376**, minus archive (41) + `apple/` (4) = 331, loader
**329 enabled** (34 hub / 52 builtin / 243 local). → [[systems/skills-index]]
- **Wiki maintenance deployed.** Quartz parsed 102 files, emitted and deployed **602 files**, and
`hermes-wiki-serve.service` restarted 00:16 AWST. Freshness: **2 stale / 94 fresh / 6 no-date of 102 pages** —
`reports/communities-wa-social-media-audit-2026.md` (56 d) plus, newly over the threshold,
`systems/model-providers.md` (22 d, updated 2026-10-01; needs a pass against the live OmniRoute catalogue).
- **27 sessions in the window:** 15 `cron`, 3 `subagent` (the AI-Newsletter research fan-out), 6 `desktop`
(voice-chat-backend research 11:28–11:32; incoming-phone-call tests 13:04 / 13:07 / 13:15), 2 `acp`
(0 messages), 1 `oneshot` memory-capture probe. Fleet unchanged at **28 jobs (25 enabled / 3 disabled)**;
**11 pin `custom:omni-101` / `smart-route`**, 8 carry no pin, 6 `free-stack`, 2 `custom:omni`, 1
`deepseek-paid-smart` (this audit).
## Recent Changes (2026-10-22, nightly audit; live probes 2026-10-21 23:59 → 2026-10-22 00:20 AWST)
- **Hermes Agent updated to `0.22.3`.** `hermes update` ran **2026-10-21 22:47:25 → 22:57:38 AWST**
(action `9eed3a99e799347c691ea52a72e122a1`), from pre-update tree `0.22.2` / `d923a5eb487a17c0…` to commit
`5b147ab036d6…` (detached, install stamp `displayVersion 0.22.3`, built 2026-10-21T14:57:38Z). Stages
`plan` / `apply` (git) / `deps` / `build` all succeeded, **but `snapshot` = `skipped`**
(`disabled by updates.pre_update_backup (mode: off)`). Gateway restarted via **systemd** — pid `868087`,
`phase: running`, `api_server: connected`, heartbeat 2026-10-22 00:13 AWST — and the dashboard was respawned
by argv on `:9119` (pid `869852`). Post-run advisories, **neither yet acted on**: rebuild the search index
(`hermes sessions optimize-storage`) and *Preparing pinned cua-driver (Computer Use)…* (no completion line in
the log). → [[systems/hermes-agent]]
- **The only cron failure of 2026-10-21 is a new failure mode.** Window: **171 `completed` / 1 `failed`** (+ this
audit). `dca8482e4f76` (**Daily Marketing Job Radar**, 07:00 weekday, `custom:omni-101` / `smart-route`) failed
at **07:40:48 after 40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by
pre-dispatch filters reset after 19m 20s` — **not** the 503 resource-pressure string (zero such rows on 10-21;
last 2026-10-19 09:02) — while the sibling 07:00 job `85edf6772bca` completed normally. Incident
`dca848_014affeb6954` **`alerted`, unclosed**; the run produced no output (`cron_incomplete_no_output`,
82 msgs / 47 tool calls). → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]
- **Incidents: 1 closed / 1 opened; unclosed stays 9** (8 `alerted` / 1 `detected`). `08eb86_472c7b609c80`
**resolved 2026-10-21 08:37:21** after the job's 08:30 slot ran clean, which also cleared `08eb86f02d37` from
the non-clean table. Non-clean `last_status` **stays 9** (7 `error` / 2 `delivery_failed`), one swap in/out.
- **2026-10-20 settled at 169 `completed` / 1 `unknown`** — the orphaned `08eb86f02d37` run. The previous
audit's "168 / 0" was taken before the 23:59 slot landed.
→ [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]
- **Config backup: 76 commits now unpushed.** The 2026-10-21 11:00 run created commit `a8486a20` and the push
failed for the 75th consecutive night; Telegram notice `message_id 8449` was delivered, AgentMail failed.
→ [[incidents/2026-08-08-gitea-config-backup-push-dead]]
- **12 sessions in the window, all `cron` — no interactive sessions.** Fleet unchanged at **28 jobs
(25 enabled / 3 disabled)**; **17 pin `custom:omni-101`**, **7 carry no pin**, 1 pins `custom:omni`.
## Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST)
- **Scheduled wiki maintenance ran clean.** `wiki-maintain.sh` completed all three steps — skills-index
regenerated (346 → **347 skills**, new `australian-tax-return-review` under Finance), freshness audit written
(`raw/stocktake/freshness-20261021.txt`), Quartz rebuilt (**596 files**) and deployed, `hermes-wiki-serve.service`
restarted. Job `1a8e90ea2271` is `active`, `last_status: ok`, pinned `custom:omni-101` / `free-stack`.
- **Tailscale node inventory corrected — the table had drifted again.** `octop` (100.92.163.122) and `springroll`
(100.100.153.125) were **missing from the table entirely**; `openmausbot` / `openmausbot-vm` appeared in prose but
not in the table body; `omarchymac` had gone **offline** (was online); every offline age was stale. Rebuilt from
`tailscale status --json` **keyed on DNS name, not `HostName`** — the JSON `HostName` for the `freshrss` node is
`ct260`, and **both OmniRoute nodes report the hostname `omniroute`**, which collides silently. Now **51 non-funnel
nodes** (50 IPv4 peers + `hermes`). → [[infrastructure/tailscale]]
- **Two Uber Eats monthly reports were missing from the published wiki and have been recovered.** The generator wrote
only to `~/wiki/reports/uber-eats/` (the `~/wiki` repo), never to the canonical `agent-estate-wiki` tree — so
**2026-08 and 2026-09 never reached the published site**. **Root cause fixed:** the generator now targets the
canonical tree and mirrors to the legacy one, and now emits **YAML frontmatter** (without it the pages landed in the
audit's "no updated DATE" bucket and Quartz could not read the page date). Both months verified against
`~/.hermes/uber_eats_report.json`; the 2026-08 previous-month column corrected from $0.00 to **$264.51 / 11 orders**.
→ [[reports/uber-eats]]
- **Two incident pages re-verified live and annotated:** [[incidents/2026-08-15-skills-index-drift]] (its named
`regenerate-skills-index.sh` no longer exists — regeneration now lives in `wiki-maintain.sh` step 1/3) and
[[incidents/2026-09-02-sftpgo-outage]] (**CT299 running**, health endpoint HTTP 401 = auth required, service up).
- **One page left stale on purpose:** [[reports/communities-wa-social-media-audit-2026]] (54d) — its source document
`/home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md` **no longer exists**, so the baseline
cannot be re-verified against a live source. Flagged, not edited.
## Recent Changes (2026-10-21, nightly audit; live probes 2026-10-21 00:03-00:15 AWST)
- **A clean day: 2026-10-20 closed 168 `completed` / 0 `failed`** (plus 1 `unknown`, plus this audit) — the first
zero-failure day since **2026-10-12**, ending a seven-day run of failures. **The `omni-101` 503 fault did not
recur** (the last failure row carrying the resource-pressure string remains 2026-10-19), and **zero new
incident rows were opened** across the whole window; 4 closed, so **unclosed rows fall 13 → 9** (8 `alerted` /
1 `detected`). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`monitoring`**)
- **A gateway unclean exit orphaned a cron run, and nothing reported it.** `08eb86f02d37` (Gmail Action Digest,
08:30 slot) was claimed 08:31:03 and closed as **`unknown`**: *“Scheduler restarted after this execution's owner
exited before a durable terminal state; whether side effects ran is unknown.”* The previous gateway life
(`pid 2496376`, started 2026-10-17 05:14) was recorded **UNCLEANLY** at 09:09:01 — last heartbeat 08:50:28, RSS
394 MB, 12.2 GB free, **`suspected_oom=False`**. `delivery_outcome` is NULL and **no delivery row exists for
the slot**, so the ledger offers no evidence the digest arrived; **no incident row was created**, and the job
table still shows the *previous* slot's 503. First recorded orphan of its kind; first unclean exit since
2026-09-21. → [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]
- **⚠ A green cron row is not evidence the work happened — second instance, and this one is 2.5 months old.**
`e27da6c34bbb` (Hermes Config Backup to Gitea, daily 11:00) reads **`last_status: ok`**, `last_error: null`, and
its 2026-10-20 execution reads `completed` — while **nothing has reached Gitea since 2026-08-07 11:00:47**.
The repo is **75 commits ahead of `origin/main`**; `backup.log` ends
`could not read Username for 'https://gitea.kangaroo-eel.ts.net'` → `✗ Push to Gitea FAILED` →
`--- Backup complete: FAIL ---`. `origin` is HTTPS with no credential in the URL and no helper, so the
non-interactive push cannot authenticate: **no off-box `~/.hermes` backup since 2026-08-08**. Telegram failure
notices have been firing all along (last 2026-10-20 11:00, `message_id 8447`); the AgentMail step failed.
→ [[incidents/2026-08-08-gitea-config-backup-push-dead]], [[infrastructure/backups]]
- **The global model default moved off OmniRoute.** `model.provider: custom` → **`nous`**
(`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` → **`meituan/longcat-2.5-preview:free`**
(not in place 2026-10-09 19:42; in place by 2026-10-20 11:00). **Pins are unchanged** — 20 of 28 jobs still pin
OmniRoute (`smart-route` ×11, `free-stack` ×6, `deepseek-paid-smart` ×1, `custom:omni` ×2) — but the **8 jobs
with no pin now call Nous direct instead of the router**, so the fleet runs two upstream paths for the first
time. → [[systems/current-profile]], [[systems/scheduled-tasks]]
- **A user question was answered with a canned non-answer, and the request is still outstanding.** Desktop
session `20261020_183748_6d40ea` (2026-10-20 18:37, `model: minimax/minimax-m3.1-flash:free`) asked *how to use
Nous OAuth dashboard auth for the gateway* and *whether a gateway token could be configured*; the turn ended
`reason=rebuilt_restart_limit_exceeded … response_len=0` and the only assistant message is *“Your request was
not processed. Send it again if you still want me to carry it out.”* **Nothing was configured in response to
that request** — it needs re-asking. → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **`dashboard.basic_auth` now carries a `secret` and `session_ttl_seconds: 43200`** (present 2026-10-21 00:0x).
The plugin's *“no 'secret' configured; generating a random per-process signing key … sessions will not survive
a restart”* warning last appeared 2026-10-20 18:45:14, so dashboard sessions should now persist across
restarts.
- **Interactive sessions returned after weeks of cron-only windows:** `talaria-ios` (10:49 → 19:27, 25 messages,
12 tool calls, *“Get Australian tax return done”*, `provider: custom:command-code`) and the `desktop` session
above. **Non-clean `last_status` 12 → 9** (7 `error` / 2 `delivery_failed`); fleet unchanged at **28 jobs / 25
enabled / 3 disabled**. → [[systems/scheduled-tasks]]
## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST)
- **The `omni-101` 503 resource-pressure fault returned 44 hours after the incident page was downgraded to
`monitoring` — same node, same error string, same blast radius.** 2026-10-19 closed **159 completed / 13
failed** against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all read
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on
`custom:omni-101`, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs, seven jobs). The
remaining 4 failed rows are the Mnemosyne Watchdog quoting the coverage monitor's error: the same derivative
double-count as burst 1. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`active`**)
- **Single pin, verified this time.** Every job that failed in burst 2 reads `provider: custom:omni-101`,
`model: smart-route` in `jobs.json` as of 2026-10-20 — there is no alias diversity left to fail over to. ⚠ The
burst-1 note recording a wider mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json`;
it needs confirming with Anthony before either reading is relied on.
- **The node stayed green through all of it.** `omniroute-1` (`100.78.63.20:20128`) answered
`/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` at 2026-10-20 00:03, before, during and after
the burst. **Reachability probes cannot see this failure class** — only job-level error text can.
- **`c0a9fa314542` (GitHub Trending Radar) lost its scheduled Monday slot** (`0 9 * * 1`, 09:01) to the 503, so
this week's roundup never ran and its revoked-token status is **untested for a second consecutive Monday**.
Its next real test is Mon **2026-10-26 09:00**.
- **Non-clean `last_status` moved 7 → 12** (10 `error`, 2 `delivery_failed`) as of 2026-10-20 00:09. Newly
dirtied: `dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`; deepened: `0e519874d280` and
`ecc1fc13655a` (both streak 2). `fad5a86979d4` added a `delivery_failed` (bare-`telegram` target, thread
34795 not found — no incident row created for it). `b63ed5817212`, `96eb95ce12f2` and `ea79a2107e36` all
cleared on their next slots. **Unclosed incident rows: 13** (12 `alerted` / 1 `detected`). → [[systems/scheduled-tasks]]
- **Unchanged otherwise:** fleet still **28 jobs / 25 enabled / 3 disabled**; CT255 Mattermost
(`100.96.100.255:8065`) still unreachable; **15 sessions in the window, all `cron` — no interactive sessions.**
## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)
- **A model stream started repeating itself — and Hermes delivered the placeholder to Telegram before it
aborted.** The FreshRSS Midday slot (`ecc1fc13655a`, 12:00) ended `failed` at 12:08:25 with
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
degenerate response.` The run had already written and **delivered** a canned line — *"Your request was
not processed. Send it again if you still want me to carry it out."* — at 12:08:11
(`delivery_outcome: delivered`), so the digest topic received a non-answer that does not read as an
error. **A delivered message is not evidence of delivered content.** Open question: should the guard
suppress delivery or re-queue, instead of substituting text on the job's normal route?
→ [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Window tally: `10-18 = 168 completed / 1 failed`** — that row is the only failure of the day;
`10-17 = 163 / 5`, and `10-19` so far `5 / 0` plus this audit. **Non-clean `last_status` 8 → 7**;
**open incidents 8 → 7** (6 `alerted` / 1 `detected`), with **1 opened and 2 resolved** in-window
(`cbfe434a3b5e` 00:42:54, `1a8e90ea2271` 03:01:22 — both auto-closed by their own next clean slot).
**11 sessions in the window, all `cron`; no interactive sessions.** → [[systems/scheduled-tasks]]
- **The 10-18 03:00 stocktake landed and closed the two-day gap** the last audit flagged: `1a8e90ea2271`
exited 0, `raw/stocktake/freshness-20261018.txt` exists, [[systems/skills-index]] was regenerated
(346 skills / 70 categories; freshness **47 stale / 41 fresh / 7 no-date**) and committed (`dd53e5f`).
- **Mattermost's cause is finally known: CT255 is stopped.** The adapter's 5-minute reconnect loop is
unchanged (`attempt 471` at 00:14:07; `mattermost connect timed out after 30s`), the peer is offline
(last seen **29d**), and the host answers neither TCP `:8065` nor ICMP from CT460. The 10-11 entry's
"cause not determined" is now superseded. → [[systems/messaging-integrations]]
- **New page: [[systems/push-notifications]] — and the push bridge is broken, and has been for a long
time.** `status.json` says `connected: false` with `connection refused (nothing listens there on this
machine)`: the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which resolves
on this host to **`127.0.1.1`** via `/etc/hosts` — where nothing serves 443 — while the gateway itself is
healthy (`0.0.0.0:9119`, `/` → 302) and the tailnet Serve route is intact. The error string appears
**23,469** times in the retained logs. Fix not applied (change the dial target, or the name resolution).
→ [[systems/push-notifications]]
- **One more standing alert, unchanged: the GitHub Trending Radar delivery token is still dead**
(`getMe` → 401, re-verified 10-18), and its next exercise is **Mon 2026-10-19 09:00**. Not repaired — a
replacement bot token is Anthony's call. → [[systems/scheduled-tasks]]
## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)
- **The `omniroute-1` node is up — and this time it shed load instead of refusing connections.** Three days
after the 34-hour TCP blackout, the same node produced a second, different failure mode: **9 model calls
across 7 jobs died with `RuntimeError: HTTP 503: Service temporarily unavailable due to resource
pressure.`** between **10-16 08:00:37 and 10-17 03:00:28** (~19 h), while `/api/health` answered
`200 {"status":"ok"}` and `/v1/models` answered `401` before, during and after (re-probed 10-18 00:33).
→ [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **It cleared itself and the fleet is green again.** `10-16 = 165 completed / 8 failed`, `10-17 = 162 / 5`
(**all five before 03:01**; a 2026-10-20 DB re-read gives **163 / 5** — one 10-17 row was counted twice at the time), `10-18` so far `7 / 0` — **no failure row of any kind since 10-17 03:00**.
Every job that failed re-ran clean on its next slot: `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17
08:31, `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, each of those auto-closing its own incident.
- **Fleet count fell 29 → 28: the Mosquito Tracker was removed.** `1b31187397a5` is absent from
`jobs.json` (file `updated_at 2026-10-18T00:36:17+08:00`); its last execution was **10-16 15:00**
(`completed`, delivered). The three disabled jobs are unchanged — **25 enabled / 3 disabled**.
- **Non-clean `last_status`: 12 → 8 — and seven of the eight cannot clear until their next slot**, because
every one of them is a weekly, Mon/Thu or monthly job: `1b23c042e17d` (`error`, streak **2**, last
failure 10-16 08:03 with the 503, next **Fri 10-23 08:00**), `0e519874d280` (Mon 10-19 08:00),
`c8d52904e1d6` (Thu 10-22 18:00), `06a0a1c8704f` (Thu 10-22 10:00), and the monthlies `0e753e9720fd`
and `066bb5191280` (both 11-01). The eighth is `cbfe434a3b5e` — this audit, whose 10-17 run died on the
503 and whose 10-18 run clears it. **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 opened and
6 closed** in-window. → [[systems/scheduled-tasks]]
- **The watchdog double-counts — 4 of the 13 failure rows are one sentinel.** `96eb95ce12f2` exited 1 on
10-16 12:15 + 12:30 and 10-17 00:15 + 00:30, each row quoting the coverage monitor's own 503 error.
Read failures per job, not per row. → [[systems/mnemosyne]]
- **Second consecutive stocktake hole.** The 10-17 03:00 slot failed (503), so
`raw/stocktake/freshness-20261017.txt` does not exist and `systems/skills-index.md` is stamped **10-16**
(10-15 is missing too, from the storm — `RuntimeError: Connection error.`). The **10-18 03:00** slot
closes it; nothing was regenerated by hand, because a manual run would race the scheduled one.
- **Fifth consecutive window with no interactive session** — **30 sessions** across the two days, all
`cron`, and **zero** non-cron sessions since 2026-10-15. Every finding here comes from live system
state, not conversation.
- **The GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in
`~/workspace/github-trending-radar/config.json` returns **HTTP 401** on `getMe` (probed 10-18 00:35);
the job's GitHub PAT still answers **200**, so only the delivery credential is gone. The cron row stays
green because `last_status` was set by the **10-12 09:00** run (`last_run_at 2026-10-12T09:00:12`).
Next exercise: **Mon 2026-10-19 09:00**. **Not repaired** — a replacement bot token is Anthony's call.
## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)
- **⚠ A fleet-wide 34-hour cron blackout just ended — `omniroute-1` was refusing TCP connections.** `10-14 = 160 completed / 12 failed` and `10-15 = 151 completed / 24 failed`, against a clean 10-10→10-12 (`175/0`, `172/0`, `175/0`) and 10-13 (`167/1`). Root cause: `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` (provider `custom:omni-101`), surfacing in the cron DB as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` Window **10-14 08:30:27 → 10-15 18:07**, recovered by **10-15 22:01**. **24 genuine model-call failures on 12 jobs** + 12 derivative watchdog reds = 36 failure rows across 13 jobs. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Everything model-calling is pinned to that one node — there is no failover anywhere in the fleet.** All **18** enabled `agent`-mode cron jobs run on `custom:omni-101` (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable pin, `custom:omni` (CT600 `omniroute:20128`), carries just the **`no_agent`** job `e530c89f2be0`, whose provider field is vestigial; `d40085da631c` is disabled. Completing runs inside the window: **211 — every single one script-only.** **Zero model calls succeeded for 34 hours.** → [[systems/omniroute]], [[systems/scheduled-tasks]]
- **The cost was silence, not errors.** Digests that never went out: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick; the weekly Perth New Restaurant was **missed outright** (next chance 10-22). The **10-14 23:59 Wiki Audit itself died on the same fault**, so the 10-14 window was never audited — this entry spans two days. → [[systems/scheduled-tasks]]
- **Recovered, verified live.** `GET /v1/models` on `100.78.63.20:20128` → **HTTP 401 in 10 ms** (up, auth required); same on `omniroute:20128` / `100.88.81.19`. First green model-call job: `1b31187397a5` at **10-15 22:01:56**; then `b63ed5817212` (**10-16 00:02:29**) and `ea79a2107e36` (**10-16 00:09:48**).
- **The backlog unwinds on schedule rather than by hand.** **Non-clean `last_status`: 3 → 12** (9 new storm rows; three cleared as their slots came round), **open incidents 3 → 12** (9 `alerted` / 3 `detected`). The two longest-lived — `1b23c042e17d` (Fri 08:00 slot) and `0e753e9720fd` (1st 09:00) — **pre-date the storm**. → [[systems/scheduled-tasks]]
- **Both Mnemosyne monitors sit inside the blast radius.** Health (12 h) and Embedding Coverage (6 h) both failed on every slot, so **no coverage figures were refreshed for ~34 h** — the last explicit read remains 2026-10-12 (working 96.7 %, episodic 99.4 %). The script-only Mnemosyne jobs were unaffected. → [[systems/mnemosyne]]
- **Self-healing gap, not a manual repair.** The failed 10-15 03:00 stocktake left `raw/stocktake/freshness-20261015.txt` unwritten and `systems/skills-index.md` stamped 10-14; the 10-16 03:00 slot closes it, so nothing was regenerated by hand here.
- **Fourth consecutive window with no interactive session** — **36 sessions** across the two days, all `cron`.
## Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST)
- **⚠ A second, unclustered Proxmox hypervisor exists — `m93p` at `100.104.51.98`, PVE 9.2.0. DISCOVERED this pass.** `pvecm nodes` is empty on both hosts (no cluster), and guest IDs **collide**: `m93p` CT221 is gitea, `pve` CT103 is cliproxyapi, and both worlds share CT100/CT221/CT245 numbers. It had been in the Tailscale node list as an unremarkable `online` peer with no wiki page recording it as a hypervisor; a tailnet-wide `:8006` scan returned only `m93p` and `pve`, which is how it surfaced. Live guests on `m93p` include gitea (CT221), ntfy (CT1241), freshrss (CT1260), openweb (CT245), vaultwarden, obsidian-livesync, rustdesk, homebridge, litellm (CT500), omniroute (CT101), paseo (CT1261), mailflow, social-dashboard, docker-01, rclone, freellmapi; `nightbridge` stopped. → [[infrastructure/proxmox]], [[infrastructure/network-map]]
- **Four guests the wiki placed on `pve` are actually on `m93p`.** CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are **absent from live `pct list` on `pve`,** yet all four services answer. `/etc/pve/lxc/` on `pve` holds only `260.conf.bak` and no vzdump log references them — so they were never deleted, **they were never on this host.** Every prior audit since 2026-10-11 carried the error forward. Always qualify a guest with its host. → [[infrastructure/proxmox]]
- **VM 282 `ubuntu-server-xfce` is running**, not stopped as recorded. → [[infrastructure/proxmox]]
- **Tailscale node inventory rebuilt: 45 non-funnel nodes** (plus 23 `funnel-ingress-node` IPv6 peers, excluded). The 2026-10-11 table had drifted badly — it listed 30 nodes and was **missing 15**, including `m93p`, `pve`, `omniroute-1`, `paseo`, `qwenpaw`, `rustdesk`, `vaultwarden`, `social-dashboard`, `openweb`, `omarchybook`, `sftpgo`, `rclone`, `omni`, `runtipi`. Several states were wrong too (`anthonys-mac-mini` is online on a different IP; `omarchy-macmini`/`omarchy-quattro` are not live peers; `anthonys-iphone` is online not active). Serve routes on CT460 re-verified **exactly** as recorded — 13 routes, `:8443/webhook` Funnel, `:9443` tailnet-only. → [[infrastructure/tailscale]]
- **Mattermost CT255 confirmed stopped, consistent** with its Tailscale peer offline 24d — unchanged finding, re-verified. → [[infrastructure/proxmox]]
- **Camofox recovered from its 2026-10-12 memory ceiling:** `camofox-browser.service` now reports `Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M)` with 16 tasks, active since 2026-10-13 23:42:55. The containment limits held; the workload was the fault. → [[systems/browser-backend]]
- **Stocktake artifacts regenerated:** `systems/skills-index.md` = **346 skills / 70 categories** (the known 347-on-disk off-by-one persists); `raw/stocktake/freshness-20261014.txt` = **48 stale / 38 fresh / 7 no-date of 93 pages**, identical in shape to 10-13.
- **Quartz rebuild still inert** (no `quartz.config.ts`, no `npx`) — `public/` was not refreshed. Carried open item.
## Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST)
- **The four-day clean streak broke: 2026-10-13 = `167 completed` / 1 `failed`** (10-12 finalised `175 / 0`, 10-11 `172 / 0`, 10-10 `175 / 0`). The failure is `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) on `RuntimeError: HTTP 404: No active credentials for provider: opencode` — the **third** failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident `1a8e90_d11698bd7275` (`detected`, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → [[systems/scheduled-tasks]]
- **The failing run stranded its own generated artifacts.** Step 1 (skills-index) and step 2 (freshness report) wrote to disk, then the agent turn died → no commit, no push. `systems/skills-index.md` (date-only change; **346 skills / 70 categories** unchanged) and `raw/stocktake/freshness-20261013.txt` (**48 stale / 38 fresh / 7 no-date of 93 pages**, down from 54 stale) were committed by the 2026-10-14 audit. → [[systems/skills-index]]
- **`executions.db` retention floor moved to 2026-10-08 07:11** (`MAX_TERMINAL_EXECUTIONS = 1000`; 1,001 rows = 1,000 terminal + 1 in flight). Days older than the floor now undercount — 10-08 reads `126 completed / 1 failed`, a partial tally.
- **Non-clean `last_status` rows: 3 → 4** (cleared `fad5a86979d4`, `dca8482e4f76`; added `1a8e90ea2271`). **Open incidents: 2 → 3.** Fleet unchanged at 29 jobs / 26 enabled.
- **No interactive sessions for the third window running:** 15 sessions in the window, all `cron`, zero interactive.
## Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)
- **A clean cron day hid a dead credential.** 2026-10-12 ran **174 completed / 0 failed** (third consecutive clean day; 10-11 finalised at `172 / 0`) — yet `c0a9fa314542` (GitHub Trending Radar) reported `completed` **with `delivery_outcome: delivered` while its own Telegram send was dying on an invalid token**. The delivered artifact was the *agent's* recovered digest, not the script's. → [[systems/scheduled-tasks]]
- **GitHub Trending Radar's bot token is REVOKED — live-verified, and the 2026-10-06 "refuted" entry was WRONG.** `getMe` against `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**; the token is a valid 46-char shape, so revoked rather than malformed, and **no valid replacement exists on the box**. The fetch path is healthy (100 repos, 26 cache files in 24 h), so only `digest.py`'s `send()` fails. **Correction:** the 2026-10-06 entry below treated two green runs as proof the token worked; a green cron row only means the *agent* finished, and neither run exercised the failing call. Fix = new @BotFather token → `config.json` → re-run `getMe`. Credential present, value not recorded. → [[systems/scheduled-tasks]]
- **Open cron incidents 5 → 2.** Closed on 2026-10-12: `fad5a8_a13bbc837e1c` + `fad5a8_c80b59a907f0` at **03:05:39** (both `deepseek-v4-flash`, cleared by `fad5a86979d4`'s first clean scheduled run — 03:00:45 → 03:05:47, 448 unique birthdays, delivered) and `dca848_23fcd2b18aaa` at **07:33:41** (cleared by `dca8482e4f76`'s own 07:00 run, delivered to topic 1030 after a 32-minute run). Survivors: `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — cannot clear until 2026-11-01, the job has **no retained executions at all**) and `1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot Fri 2026-10-16 08:00). Table: alerted 1 / detected 1 / closed 8 / resolved 66. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` **5 → 3** (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`). → [[systems/scheduled-tasks]]
- **Camofox hit its memory containment ceiling inside a cron run (NEW).** The 2026-10-12 07:01 Daily Marketing Job Radar drove `camofox-browser.service` to **`2.5G` with swap fully consumed, `available: 0B`, 383 tasks** (drop-in `MemoryHigh=2G`/`MemoryMax=3G`/`MemorySwapMax=1G`); `/tabs` and page evals hung. The agent restarted the service at **07:23:50** and finished **one tab at a time**, and the job still delivered. A green cron row does not mean the browser layer was healthy — check the unit's peak/swap/tasks when a browser-heavy job runs long. → [[systems/browser-backend]]
- **Browser page corrected against a live probe.** REST API is **`:9377`**, not `:9093` (nothing listened on 9093); `:9378` does not exist; Camofox is a `systemd --user` unit **on the Hermes host (CT460)**, not CT450. Containment drop-in documented. → [[systems/browser-backend]]
- **Mnemosyne healthy at 12:02** — working **760/786 = 96.7 %**, episodic **1050/1056 = 99.4 %** (both above the 95 % floor, no `reindex` needed); orphan embeddings 0; FTS5 aligned exactly (786/786, 1056/1056, 876/876); 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → [[systems/mnemosyne]]
- **Window shape:** **18 sessions, all `cron`, zero interactive** — the other bookend to the previous window's `talaria-ios` research session. The 03:00 stocktake `1a8e90ea2271` ran clean and pushed **`0eee5f1`** (freshness **54 stale / 32 fresh / 7 no-date of 93 pages**; skills index 346 / 70 categories — the known 347-vs-346 off-by-one persists). The execution log's retention floor has moved to **2026-10-07 08:15**, so 10-07 rows are now pruned.
## Recent Changes (2026-10-12, 03:00 stocktake; live probes 2026-10-12 03:0x-03:1x AWST)
- **02 stocktake corrections from live probes.** [[systems/nanobot]] was DOWN as of 2026-08-07; its web UI on CT333 is **back up** (HTTP `200` on `100.122.189.81:8766`), though the OmniRoute port `8900` endpoint is still closed so `status` stays `degraded`. [[systems/home-assistant]] confirmed **up**: HTTP `200` on `100.68.139.5:8123`, VMID 100 running, Tailscale node offering an exit node — confidence raised low → medium. → [[systems/nanobot]], [[systems/home-assistant]]
- **Freshness snapshot: 54 stale / 32 fresh / 7 no-date of 93 pages** (threshold >21 days). The bulk of the stale list is the 2026-07-22 seeding cohort (concepts, runbooks, comparisons) and is structural, not a regression. → `raw/stocktake/freshness-20261012.txt`
- **Skills index regenerated: 346 skills / 70 categories**, unchanged count from the previous run. → [[systems/skills-index]]
- **Tailscale routes re-verified** against `tailscale serve status` — the CT460 table in [[infrastructure/tailscale]] matches live output exactly (13 tailnet routes + `:8443/webhook` Funnel + `:9443` Vite dev). Two nodes joined since 2026-10-11: `omarchy` (100.73.227.103) and `omniroute-1` (100.78.63.20, both active).
## Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST)
- **The cron fleet is clean for a second consecutive day: 2026-10-11 ran `171 completed / 0 failed`** (+1 `running` = this audit), after 10-07 `1` / 10-08 `2` / 10-09 `15` failed. 10-10 finalises at **`175 / 0`**. Fleet **29 jobs — 26 enabled / 3 disabled**; non-clean `last_status` steady at **5**; **0 incidents opened, 0 closed — 5 unresolved** (3 `alerted`, 2 `detected`). The two streak-1 jobs (`dca8482e4f76`, `1b23c042e17d`) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → [[systems/scheduled-tasks]]
- **The 2026-10-11 00:08 "memory capture stalled" EXCEPTION was a timezone mislabel, not a capture fault.** `working_memory.created_at` is naive UTC; the monitor compared it against AWST `now` and reported a 2.7 h gap as 10.7 h. Capture verified healthy: 786 rows, +2 in the window, coverage 98.3 % working / 99.6 % episodic. → [[systems/mnemosyne]]
- **Proxmox storage documented for the first time:** `/mnt/ssd0` is the pressure point — 808 GiB used, **~63 GiB free (93 % by `df`, 88.1 % by `pvesm`)**; `SSD1` 86.1 %, `fred-qnap` 76.8 %, `iMacHDD` inactive. → [[infrastructure/proxmox]]
- **First retained research answer filed:** fastest VNC clients (asked 2026-10-11) — the server-side encoder is the real bottleneck, TigerVNC native viewer for standard RFB, `-CompressLevel` is a *client* parameter, RealVNC's "4.7×" is self-referential. → [[queries/retained-research-answers]]
- **Only one interactive session in the window** (`talaria-ios`, 19:13 → 19:51, 41 messages); the other 17 were `cron`. The 03:00 stocktake `1a8e90ea2271` completed on schedule (03:00:40 → 03:11:29) and committed `d981e30`.
- **Stale labels corrected:** [[incidents/2026-08-18-mnemosyne-capture-pipeline]] moved from `active` to `superseded` (evidence above); the index's count of open 2026-10-09 storm incidents trimmed to the 2 that are genuinely open.
## Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST)
- **OmniRoute CT600/CT601 roles are INVERTED.** CT600 `omniroute` (100.88.81.19) is **running** and serving `:20128` (combo API returns live combos); CT601 `omni` (100.93.204.4) is **stopped** with `onboot=0` (Tailscale peer offline 54d). Hermes resolves `custom:omni` → `omniroute:20128` → **CT600**. → [[systems/omniroute]]
- **A third OmniRoute node exists:** `omniroute-1` at `100.78.63.20:20128`, live (401 = auth-required), registered in Hermes config as provider `omni-101` (5384 models, default `smart-route`). It carries `skills_hub` (model `free-stack`) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in `pct list`; identity unconfirmed. → [[systems/omniroute]]
- **Two Ignis instances exist, on different hosts.** CT460 `ignis` (port 8080, vault `/home/hermes/wiki/agent-estate-wiki`, serving 200) and CT200 `ignis-ignis-1` (image `ignis-server:latest`, host port 8092, vault `/opt/ignis/vaults`, serving 200). Both healthy; they are separate deployments. → [[systems/ignis]]
- **Tailscale route drift fixed:** `/agent-estate` targets bare `http://127.0.0.1:8769` (page had `.../agent-estate-role-dashboard.html`); a tailnet-only `:9443` → `127.0.0.1:5173` Serve was undocumented. CT201's live routes are 8085-based plus `/steel` (5173) and `/steel-api` (3000) — the old `:8443` → `localhost:8082` Homepage row is wrong. *(The `/steel` and `/steel-api` routes were deleted on 2026-10-29 — see [[systems/browser-backend]].)* → [[infrastructure/tailscale]]
- **Proxmox inventory rebuilt** from `pct list` + `qm list`: **CT475 `agentzero` and VM110 `omarchy` were missing entirely**; 282 `ubuntu-server-xfce` corrected to stopped; dead rows removed (310, 500, 104, 106, 101). **VM403 `open-claw` and VM401 `hermes-old` are absent from `qm list`** — flagged unconfirmed, not deleted. → [[infrastructure/proxmox]]
- **Nanobot verified on CT333:** `nanobot-gateway` (PID 2259048) alive on 8765/8766. It is **not** a systemd `nanobot.service`/`gateway.service` (both probes return `inactive` — wrong unit names); it runs as the `nanobot` user service plus the gateway process. Port **8900 is no longer listening** — previous "Serve API" claim flagged unverified. → [[agents/nanobot]]
- **Docker services page corrected:** the old "Docker runtime not available" line was a CT460 socket artifact (`/var/run/docker.sock` is `root:docker srw-rw----`; `hermes` user not in group `docker`; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → [[infrastructure/docker-services]]
- **Mattermost CT255 confirmed stopped**, consistent with its Tailscale peer being offline 21d. Root cause of the unreachable endpoint still undetermined.
- **⚠ `mycompose-db-1` on CT200 is in `Restarting (1)`** — crash-looping. Flagged; root cause NOT investigated this pass.
- Stocktake artifacts: skills-index regenerated (**346 skills / 70 categories**), `raw/stocktake/freshness-20261011.txt` = **62 stale / 24 fresh / 7 no-date of 93 pages**.
- **Quartz rebuild still inert** (no `quartz.config.ts`) — `public/` was not refreshed. Carried open item; unchanged from prior days.
- 62 pages remain >21d stale and were **not** corrected this pass: they were not probed against a live source, so they stay flagged rather than rewritten.
## Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST)
_The manual 2026-10-10 stocktake entry follows beneath this one._
- **The cron fleet is clean again: 174 `completed`, 0 `failed` on 2026-10-10** (AWST day, `executions.db`) - after 10-07 `1` / 10-08 `2` / 10-09 `15` failures. 7 of the 10 storm incidents auto-closed during the day; **5 remain open** over 3 jobs (`1b23c042e17d` + `dca8482e4f76` `timeout`/`alerted` since 2026-10-09; `0e753e9720fd` + `fad5a86979d4` `agent`). Non-clean `last_status` down from 10 to **5**. Job count **30 -> 29 (26 enabled / 3 disabled)**. -> [[systems/scheduled-tasks]]
- **The DHL monitor `62619aa6bcb3` is retired - open item #1 closed.** Anthony instructed removal again on the evening of 2026-10-10; the job is **absent from live `jobs.json`** (verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID.
- **The 2026-09-28 runtime-dependency class recurred - in the *gateway* env this time.** The gateway's `activate_dependencies()` loads one committed uv env (`795020dbf24343cd9c52df24723dca83`); it was missing `mnemosyne` / `mnemosyne_hermes` plus native deps, and `elevenlabs` was missing from the live managed env. Repaired 21:26-21:34, verified live from **22:01:19** (`Memory provider 'mnemosyne' registered (40 tools)` -> `activated`; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does **not** cover this env - open. -> [[incidents/2026-09-28-hermes-managed-runtime-deps]]
- **New page: [[systems/paseo]]** - Paseo coding-agent daemon, user unit `paseo-daemon.service`, v0.11.2, `0.0.0.0:6767`, running since 2026-10-10 14:47:28. Pitfalls recorded: `daemon run` vs `daemon start`, `/usr/bin/paseo` is 0.7.2 while the daemon is 0.11.2, and a truncated `cline` binary presenting as a provider fault.
- **Mattermost unreachable from CT460** - configured endpoint `100.98.116.14:8065` gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> [[systems/messaging-integrations]]
- **Health-monitor follow-ups (2026-10-10):** "capture silent" was a false alarm (cron-only window); `check_embedding_coverage.py` now carries mode `0755`; 13 orphan `memory_embeddings` rows (oldest 2026-08-22) remain unverified. -> [[systems/mnemosyne]]
## Recent Changes (2026-10-10)
- **The 2026-10-09 timeout storm is over — the fleet is green again.** Window tally (`executions.db`, 2026-10-09 00:00 → 2026-10-10 00:00 AWST): **163 `completed`, 15 `failed`**. Every failure was `RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, confined to 00:00 → 15:00; all runs from 18:00 onward completed. **10 open incidents remain** (`ecc1fc_58b6c38296f3`, `08eb86_634f1a296f78`, `1b23c0_c35703f637b8`, `85edf6_18bedbeec5cd`, `dca848_23fcd2b18aaa`, `1a8e90_dc6b2d40ec44`, `1a8e90_789224ad589c`, `0e753e_e7f8194a7d06`, `fad5a8_c80b59a907f0`, `fad5a8_a13bbc837e1c`). → [[incidents/2026-10-09-cron-provider-timeout-storm]]
- **`1a8e90ea2271` (stocktake) is running *right now*, not failed.** Its 2026-10-09 03:00 run timed out before the generator produced output (so no `freshness-20261009.txt` exists — the two-day artifact gap stands); the 2026-10-10 03:00:17 run is `running` at audit time. `failure_streak: 2` until it completes. This stocktake (run manually by this audit) regenerated `systems/skills-index.md` (**346 → 346 skills / 70 categories**, frontmatter dates refreshed) and wrote `raw/stocktake/freshness-20261010.txt` — **62 stale, 23 fresh, 7 no-date of 92 pages**. See [[systems/scheduled-tasks]].
- **Skills count reconciled:** 347 `SKILL.md` on disk vs 346 reported — unchanged off-by-one (the generator de-dups the nested `productivity/obsidian-git-sync/obsidian-git-sync/` duplicate). Carried open item.
- **8 jobs carry a stale `last_status: error`** — `0e753e9720fd` (streak 0), `fad5a86979d4` (streak 0), `dca8482e4f76` (streak 1), `1b23c042e17d` (streak 1), `85edf6772bca` (streak 1), `08eb86f02d37` (streak 1), `ecc1fc13655a` (streak 1), `1a8e90ea2271` (streak 2). Two are **pre-repin** stale fields (streak 0), not live failures. Job count unchanged: **30 (27 enabled / 3 disabled)**.
- **DHL monitor `62619aa6bcb3` still enabled** — re-probed 2026-10-10: `enabled: True`, `paused_at: null`, schedule `0 8,11,14,17,20 * * *`. Anthony instructed its deletion on 2026-10-09 21:24 and a session reported it deleted. The parcel (`8052838382`) was `Delivered` 2026-10-07, so the monitor is redundant. **This audit did not mutate cron state** — retirement is open item #1 in [[systems/scheduled-tasks]].
- **Quartz rebuild step confirmed still inert** (no `quartz.config.ts`, `npx` present but unreachable for `quartz`) — the `public/` static site was **not** refreshed by this stocktake. Skills-index and freshness artifacts regenerated and committed correctly; only the published-HTML step is a no-op. Carried open item from 2026-10-07.
- **Memory recall ranking fix still not persisted** — the 2026-10-09 in-session "config saved" claim is not in live config; carried open item. Capture/vector health unchanged (760/760 working, 1,043/1,047 episodic embedded).
- **`infrastructure/tailscale.md` re-verified 2026-10-10** — `tailscale serve status` matches the page exactly: 13 tailnet Serve routes (`/`, `/ui`, `/wiki`, `/ignis`, `/webui`, `/gitea`, `/jobs`, `/desktop`, `/vnc-camofox`, `/synergy-prep`, `/agent-estate`, `/evening-grid.html`, `/ignis`/`:8443` funnel `/webhook`) + the `:8443/webhook` public Funnel. No drift; page remains fresh (19d).
## Recent Changes (2026-10-09)
- **Cron fleet: the worst day since the repin.** 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; `1a8e90ea2271` sits at `failure_streak: 2`. → [[incidents/2026-10-09-cron-provider-timeout-storm]]
- **The 2026-10-08 audit window was lost** to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → [[log]]
- **`62619aa6bcb3` (DHL Tracking — Parcel 8052838382) is still enabled** even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in [[systems/scheduled-tasks]].
- **Memory recall relevance incident** — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → [[incidents/2026-10-09-memory-recall-relevance]]
- **Mnemosyne live counts (2026-10-09 22:27):** 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → [[systems/mnemosyne]]
## Recent Changes (2026-10-08)
- **Cron fleet green for a third consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters still read 0. Window tally (`executions.db`, 2026-10-07 00:00 → 2026-10-08 00:00 AWST): **176 `completed`, 1 `failed`, 1 `running`** (the nightly audit itself). 13 sessions started in the window, all `source = cron` — **no interactive (non-cron) sessions**; 12 deliveries, **0 delivery failures**.
- **First detect → notify → auto-resolve incident cycle since the managed-runtime event (DHL, 2026-10-07 08:00 → 11:01).** `62619aa6bcb3` (DHL Tracking, 5×/day) failed its 08:00 run inside `ensure_browser()` with `TimeoutError: timed out` — Camofox briefly unresponsive on a browser HTTP call (`dhl_tracking_monitor.py:233`). The failure notice went out at 08:00:58 and incident `62619a_f485ccfd4a0d` (`failure_type: timeout`) **auto-resolved at 11:01:13** when the next 3-hourly run completed; the 14:00, 17:00 and 20:00 runs also succeeded. No action needed — and the incident machinery behaved exactly as designed, which it had not been tested on since 2026-09-30. See [[systems/scheduled-tasks]].
- **The tracked parcel arrived — the DHL monitor is now redundant.** The 14:00 run on 2026-10-07 reported waybill `8052838382` as **`Delivered`**. `62619aa6bcb3` still runs 5×/day for no remaining benefit; flagged as a removal candidate rather than removed.
- **The stocktake job proved its repin for a second day.** `1a8e90ea2271` completed automatically at **2026-10-07 03:00** — 346 skills / 70 categories (unchanged), `raw/stocktake/freshness-20261007.txt` (62 stale, 21 fresh, 7 no-date of 90 pages), committed to Gitea as `8c63ef2`. Two script-level findings came out of the run: (1) the **Quartz rebuild step is a no-op** — no `quartz.config.ts` is present and `npx` is unavailable, so the published site is never refreshed; (2) `skills-index` **under-counts by one** — 347 `SKILL.md` files exist on disk while the page reports 346, because the generator de-dups the nested `productivity/obsidian-git-sync/obsidian-git-sync/` duplicate away. Both are now recorded as open issues in [[systems/scheduled-tasks]].
- **Nothing else changed in the window.** The three stale pre-repin `last_status: error` jobs (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) remain — due Fri 08:00 / Mon 03:00 / 1st 09:00. The `custom:omni` stale-key references and the three bare-`telegram` targets are unchanged. No new incident page was required, and no page was removed or archived.
## Recent Changes (2026-10-07)
- **Cron fleet green for a second consecutive day — 0 failing.** Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 `failure_streak` counters read 0. Window tally (`executions.db`, 2026-10-06 00:00 → 2026-10-07 00:00 AWST): **176 `completed`, 0 `failed`, 1 `running`** (the nightly audit itself) — 177 rows. 14 sessions started in the window, all `source = cron`; **no interactive (non-cron) sessions**. **No new incident was opened.** The repin wrap-up closures at 00:10–00:12 on 2026-10-06 fall inside this window but were already recorded by the 2026-10-06 audit.
- **The stocktake job proved its repin — first fully automatic run since 2026-09-25.** `1a8e90ea2271` (`wiki-stocktake-maintenance`, 03:00 daily) ran on schedule **2026-10-06 03:00:31 → 03:03:32** on its new `custom:omni-101` / `free-stack` pin and completed cleanly. Its four HTTP-410 incidents (`6df957a05ec6`, `d284168d3663`, `cda67de5b3e3`, `e31584909bd2`) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. It regenerated `systems/skills-index.md` (**344 → 346 skills**), wrote `raw/stocktake/freshness-20261006.txt` and committed to Gitea as `33f4e95`. Its stale pre-repin `last_status: error` is now cleared — live `last_status: ok`, streak 0. **The daily manual `wiki-maintain.sh` run is no longer needed.** See [[systems/scheduled-tasks]].
- **Three jobs still carry a stale pre-repin `last_status: error`** — `1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd` (down from four). Each clears on its next scheduled slot (Fri 08:00 / Mon 03:00 / 1st 09:00); none is a live failure. The bare-`telegram` target warning still stands on `066bb5191280` and `0e753e9720fd` (unchanged).
- **Wiki vault mirror refreshed.** `systems/skills-index.md` in `/home/hermes/wiki/agent-estate-wiki/` had drifted to 2026-10-03 / 344 skills while the canonical copy regenerated to 2026-10-06 / 346; this audit re-mirrored it along with the pages below. The two-wiki-tree drift remains a standing open item.
## Recent Changes (2026-10-06)
- **Dead model pins cleared across the whole fleet — failing set 7 → 0.** All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: `cliprox/terra` (HTTP 401 connection auth, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`). They were moved to `custom:omni-101` on **routing aliases** — `free-stack` ×6, `smart-route` ×1 — because the fault was raw-pin brittleness, not any one model. Every `failure_streak` in `jobs.json` now reads 0. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale `last_status: error` until their next scheduled slot. See [[incidents/2026-10-05-dead-model-pin-repin]].
- **The `cliprox/terra` connection incident is closed for cron purposes.** No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- ~~**GitHub Trending Radar's revoked-token diagnosis is refuted.**~~ **SUPERSEDED 2026-10-13 — this conclusion was WRONG.** `c0a9fa314542` ran on schedule at **2026-10-05 09:00 and completed cleanly** — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No `getMe` → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
- **A stale failure notice, not a live outage, opened the review.** Anthony acted on a `85edf6772bca` notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.**
- **Window tally** (`executions.db`, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 `completed`, 3 `failed` (all pre-repin, 03:00–07:00), 2 `unknown` (aborted first attempts of `79f9bb41ecf5` / `dca8482e4f76`, superseded by their successful re-runs) and 1 `running` (the nightly audit itself). No interactive sessions other than the repin session.
## Recent Changes (2026-10-05)
- **Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7.** Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is `79f9bb41ecf5` (Weekly Review, Sun 18:00), which ran on schedule **2026-10-04 18:00** and failed with the *identical* `cliprox/terra` error the incident predicted: `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That is the last untested job on the pin, so the provider-connection auth expiry is now **3 of 3 confirmed** — settled as a credential fault, not per-job. No review was produced; `delivery_outcome: delivered` again because the failure notice is the deliverable. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **`1a8e90ea2271` failed again — 2026-10-04 03:00 (HTTP 410).** The `wiki-stocktake-maintenance` job remains pinned to the Ollama-retired `qwen3.5:397b`; no change, still needs a repin. Its script is healthy. See [[systems/scheduled-tasks]].
- **No interactive (non-cron) sessions in the window** — all 10 sessions active in the past 24h were cron. The only durable changes are cron-fleet status, reflected above.
- **Streak-number caveat for future audits:** the `N failures in a row` figure in `hermes cron list` is the `failure_streak` counter from `jobs.json`, not a DB-derived count. `~/.hermes/cron/executions.db` now retains only rows from **2026-09-30 onward** (1,001 rows), so DB recounts undercount older streaks. Live counters 2026-10-05: `1a8e90ea2271` ×11, `dca8482e4f76` ×7, `fad5a86979d4` ×3. Quote the live counter, and note the retention floor.
## Recent Changes (2026-10-04)
- **Both Hermes instances updated on 2026-10-03, from opposite directions.** CT460 (production) was rebuilt to `v0.21.5+6169.gab86a20` (upstream `ab86a20a`, tree mtime 10:11 AWST) and CT450 (chaos) came up from `v0.19.0` to `v0.21.5+6170.ga510d6c` (upstream `a510d6c6`, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports `Update available: 3 commits behind`. CT450 has no SSH — it is reached via `ssh root@100.96.100.82` + `pct exec 450`, and `hermes` is not on the default `pct exec` PATH. See [[incidents/2026-10-03-ct450-hermes-update]].
- **Vector recall had been silently running keyword-only — found and fixed 2026-10-03.** `fastembed` was missing from every uv-managed runtime, so the dense leg of `mnemosyne_recall` contributed no candidates and recall fell back to FTS5 **without erroring**. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, `FASTEMBED_CACHE_PATH` pinned, and `hermes-runtime-deps-guard.sh` extended to assert fastembed + cache path + model dir every 15 min. **Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%)**, 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See [[incidents/2026-10-03-mnemosyne-vector-recall-fts-only]] and [[systems/mnemosyne]].
- **Managed runtimes pruned 10 → 3.** Only 3 uv-managed venvs remain under `~/.hermes/installs/019d0d114d22b7e6/environments/` (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
- **Cron fleet re-verified 2026-10-04 00:12 AWST:** 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. `79f9bb41ecf5` — the last untested `cliprox/terra` job — is due 2026-10-04 18:00 AWST. See [[systems/scheduled-tasks]].
- **Mattermost is unreachable from CT460 right now** (`mattermost connect timed out after 30s, next retry in 300s`, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
- **Wiki maintenance gotcha worth remembering:** `mnemosyne doctor` runs under the legacy Python 3.11 `venv/`, where `sqlite_vec` is not importable, so it prints `vec0: present_but_unloadable` and marks vector tiers `unavailable` *even when the gateway is embedding fine*. Judge vector health with `check_embedding_coverage.py`, never with doctor's `vec0` line.
## Recent Changes (2026-10-03)
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
- **Streaks recounted from `executions.db` — one number corrected:** `dca8482e4f76` 6 → **7** (one more daily run), but `1a8e90ea2271` **8 → 7** — the ×8 carried since 2026-10-01 was an overcount; that job has exactly 7 recorded runs and all 7 failed. Recount before quoting a streak.
- **A delivered notice is not a successful job.** Both `cliprox/terra` failures are recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable — the execution `status` is `failed` and no digest/plan was produced. Judge cron health on the `status` column, not the delivery outcome.
- **Resolved — `1b23c042e17d` stale `delivery_failed` flag cleared:** the 2026-09-30 retarget to `telegram:-1004321904721:102` worked exactly as predicted; its 2026-10-02 run delivered (then failed on the model pin). The prediction held.
- **[[systems/skills-index]] regenerated by hand: 341 → 344 skills.** The daily stocktake job (`1a8e90ea2271`) is still blocked on a retired model (HTTP 410, `qwen3.5:397b`), but its **script is healthy** — `wiki-maintain.sh` was run manually on 2026-10-03 and completed cleanly, refreshing the index and writing `raw/stocktake/freshness-20261003.txt`. One new skill appeared: `fleet-wide-failure-triage` (devops). Only the scheduled wrapper is broken; every non-model step of the stocktake now needs a manual run each day until it is repinned.
- **Work mailbox blocker documented — and it is older than it looks.** A new incident page records that the `work` Outlook account has been unreachable since **2026-08-26**, not this week: the account uses password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. The TCP/TLS path is fine — it is a tenant policy wall, not a bad password, and re-running the credential helper can never fix it. It needs an OAuth2 path. No cron job polls that mailbox, so this is an on-demand capability gap. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
- **GitHub Trending Radar token claim downgraded to "unverified":** the revoked-bot-token (`getMe` → 401) diagnosis could not be reproduced — `c0a9fa314542`'s last recorded run (2026-09-30 19:22, off-schedule) **completed and delivered**. Re-confirm at the next scheduled run (Mon 2026-10-05 09:00) before acting on it.
## Recent Changes (2026-10-02)
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **3 → 6**: two *new* failures landed on 2026-10-01 —
- `06a0a1c8704f` (AI Newsletter Curation for Patrick, Thu 10:00) — `HTTP 401 … All 1 connection(s) authentication expired — please reconnect in the dashboard` on provider `custom:Omniroute:20128`, model `cliprox/terra`. Its failure notice reached topic 346; no digest was produced. **Not self-recovering** — needs the OmniRoute connection re-authenticated. Two other enabled jobs share that pin (`1b23c042e17d`, `79f9bb41ecf5`) and will fail next run.
- `0e753e9720fd` (Facebook Birthday Monthly Scrape, 09:00) — `HTTP 400`, `nvidia/deepseek-ai/deepseek-v4-flash` absent from the active NVIDIA catalog, confirming the earlier diagnosis that this is a **shared-pin** problem (sibling `fad5a86979d4` uses the same model).
- `066bb5191280` (Uber Eats Monthly Spend Report, 02:11) — report generated correctly; **delivery** failed with `configured thread_id 34795 for telegram:1793951355 was not found`.
- **Carried-over failures, streaks advanced:** `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` → streak 6; `1a8e90ea2271` HTTP 410 (retired `qwen3.5:397b`) → streak 8; `fad5a86979d4` HTTP 400 → streak 3. All three still need a model repin.
- **Telegram delivery semantics corrected — a "failed" delivery is not always a lost message.** Against `~/.hermes/cron/executions.db`, the 2026-10-01 failures carry the *fallback* variant: `configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id`. Content reaches the DM, yet the run is recorded `delivery_outcome: failed`. The 2026-09-30 incident's `Message thread not found` variant is the one that truly loses the message. Both variants are now documented in [[systems/messaging-integrations]].
- **The wiki has two trees, and they have drifted in both directions (new finding, 2026-10-02):** the canonical source is **`/home/hermes/workspace/agent-estate-wiki`** — it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build (`serve.py`, `package.json`, `public/`) and is what `wiki-maintain.sh` commits. **`/home/hermes/wiki/agent-estate-wiki`** is the *Obsidian/Ignis vault copy* (pages nested under `wiki/`, no git remote) that the iPad reads. The nightly audit is told to write to the vault path, so edits that skip the canonical tree never reach Gitea or the published site. Worse, the trees have diverged backwards too: `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist **only in the vault**, while `raw/stocktake/*`, `incidents/2026-09-02-sftpgo-outage.md`, `reports/communities-wa-social-media-audit-2026.md` and `agents/` exist **only in canonical**. `index.md` in canonical links `[[systems/ignis]]` and `[[systems/nextcloud]]`, both of which are broken there. Any one-way sync will silently delete pages — reconcile before trusting either tree. See [[systems/scheduled-tasks]].
- **Skills inventory:** 344 `SKILL.md` under `~/.hermes/skills/` (live count 2026-10-02), against 341 in the last generated [[systems/skills-index]] (2026-09-27). The page cannot refresh while the stocktake job stays broken.
## Recent Changes (2026-10-01)
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **4 → 3**: `cbfe434a3b5e` returned to `ok` as predicted on 2026-09-30. The residual `1a8e90ea2271` dispatch error also cleared — its current failure is a different, upstream problem (below).
- **Telegram delivery failures root-caused and fixed:** the target, not the topic, was wrong — a bare `deliver: "telegram"` resolves to the **home DM (`1793951355`)**, where group topic IDs cannot exist, so sends hard-failed and the message was lost. `1b23c042e17d` and `c8d52904e1d6` retargeted to `telegram:-1004321904721:102` and verified. Three jobs still carry bare `telegram` targets (`0e753e9720fd`, `fad5a86979d4`, `066bb5191280`). See [[systems/messaging-integrations]].
- **Three jobs pinned to models that no longer exist upstream — cannot self-recover:** `1a8e90ea2271` → `qwen3.5:397b` retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`, streak 5); `fad5a86979d4` → `nvidia/…/deepseek-v4-flash` absent from the live catalog (HTTP 400, streak 3). Repin to the `smart-route` alias. See [[systems/model-providers]].
- **GitHub Trending Radar — revoked Telegram bot token:** `c0a9fa314542` builds its digest then fails the push (`getMe` → 401). Token in `/home/hermes/workspace/github-trending-radar/config.json` needs replacing from @BotFather.
- **Memory hygiene pass (2026-09-30):** 36 gateway background-process notifications had been written into Mnemosyne as `[USER] [Anthony Martin]` memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts `hermes_cron_model_default` and the job-radar model corrected. See [[systems/mnemosyne]].
- **Provider stack rebuilt from live config:** 5 custom providers (`omni-101`, `Omniroute:20128`, `litellm`, `command-code`, `guacano`). The provider key `custom:omni` **no longer exists**; two cron jobs (`e530c89f2be0`, `d40085da631c`) still name it but have not failed, because the default `custom` block resolves to the same base URL.
- **Vision routing — request recorded, not applied:** Anthony asked on 2026-09-30 that vision calls use the local Ollama at `http://anthonys-mac-mini:11434` (gemma4). Live config still points at `nous / stepfun/step-3.7-flash:free` as at 2026-10-01.
## Recent Changes (2026-09-30)
- **Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST:** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause: the uv-managed runtime venvs (10 under `~/.hermes/installs/019d0d114d22b7e6/environments/*/venv`) were missing `dotenv` and `mnemosyne`, so the worker died on import before acknowledging the job (`ModuleNotFoundError: No module named 'dotenv'` — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2 (checks/repairs **all** managed venvs via `pip install --target`), run every 15 min by job `96eb95ce12f2`. `weekly-save-restricted-bot-restart` script restored 19:17; that job is `ok` again. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
- **Hermes upgraded since last audit:** v0.19.0 (2026.7.20) → **v0.21.5+3837.gbfda74c (2026.9.24)**, upstream `27062c34`, update available. Skills 330 → 341.
- **Provider routing drift:** default provider is `custom` → `http://100.78.63.20:20128` (OmniRoute) with default model `smart-route`; auxiliary curator `openai-codex / gpt-5.6-terra`. Verified live 2026-09-30.
- **Open — Daily Marketing Job Radar blocked:** job `dca8482e4f76` pinned to `custom:omni-101` / `gpt-5.6-terra`; upstream `command-code` returns `403 MODEL_NOT_IN_PLAN`. 5 consecutive failures (2026-09-26 → 30). See [[runbooks/job-radar]].
- **Open — Telegram thread delivery failures (root cause found 2026-10-01):** `telegram:1793951355` thread `34795` "not found" for jobs `fad5a86979d4` and `1b23c042e17d`; `c8d52904e1d6` reports "Message thread not found". Cause was the delivery target, not the topic — a bare `telegram` resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
- **Wiki tree structure confirmed (drift unresolved):** `/home/hermes/workspace/agent-estate-wiki` is the Gitea/Quartz source (this repo); `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault (pages under `wiki/`, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
- **Multi-bot group chat question raised:** Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
## Recent Changes (2026-09-02)
- **SFTPGo outage resolved (02:18 AWST):** Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's `tailscaled` had died, breaking tailnet reachability; PVE's `pct start` then failed because 5 stale `lxc-attach` probes from earlier diagnostics pinned CT299's cgroup (`/sys/fs/cgroup/lxc/299/ns/.lxc`). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), `pct stop 299 --skiplock` + `pct start 299`, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: `tailscale ping sftpgo` → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner `SSH-2.0-SFTPGo_2.7.0`. See [[incidents/2026-09-02-sftpgo-outage]].
## Attested facts
- **Config version:** `_config_version: 46` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously recorded as 33 on 2026-07-25)
- **Default profile:** `default` — Verified, WebUI prompt, 2026-07-22
- **Timezone:** `Australia/Perth` — User-provided + Verified in config, 2026-07-22
- **Main provider:** `custom` → `http://100.78.63.20:20128` (OmniRoute) — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `custom:litellm` → `litellm:4000`, then `custom:omni` → `omni.kangaroo-eel.ts.net`)
- **Default model:** `smart-route` — Verified, `~/.hermes/config.yaml`, 2026-09-30 (previously `freellmapi-auto`)
- **Fallback providers:** `opencode-go / mimo-v2.5`, `opencode-zen / deepseek-v4-flash-free`, `nous / stepfun/step-3.7-flash:free`, `ollama-cloud / qwen3.5:397b`, `kilocode / poolside/laguna-m.1:free`, `custom / litellm:4000` — Verified, config, 2026-09-30
- **Auxiliary curator model:** `openai-codex / gpt-5.6-terra` — Verified, config, 2026-09-30
- **Memory backend:** Mnemosyne — Verified, config, 2026-07-22
- **Memory enabled:** true — Verified, config, 2026-07-22
- **Streaming enabled:** true — Verified, config, 2026-07-22
- **Dashboard public URL:** `https://hermes.martinwa.org/` — Verified, config, 2026-07-22
- **Pet config:** enabled/slug null — Verified, config, 2026-07-22
- **WebUI connected platforms:** local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
## Recent Changes (2026-08-02)
- **Browser backend reverted:** Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. `browser.cloud_provider: camofox`, `camera_provider: camofox`. Steel plugin disabled *(the 2026-08-02 state — the plugin and the Steel deployments were later removed entirely on 2026-10-29, see [[systems/browser-backend]])*. See [[comparisons/steel-vs-camoufox]] and [[decisions/rollback-steel-to-camofox]].
- **Mnemosyne capture fix (2026-08-01):** Automatic conversational capture was broken by stringified YAML (`sync_roles: "['user']"`). Fixed by rewriting structured Mnemosyne settings with native YAML types. See [[incidents/2026-08-01-mnemosyne-capture-failure]].
- **Mnemosyne vector coverage gap discovered:** Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
- **DeepSeek V4 Flash 0731:** Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (`^[a-zA-Z0-9_-]+$`) — may cause issues with OmniRoute-transformed tool names.
- **Uber Eats report upgraded:** Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with `camofox-browser-automation` + `browser-backend-verification` skills.
- **GitHub Trending Radar — Daily Digest removed:** Stopped by user on 2026-08-02. Weekly roundup still active.
- **Ops profile renamed:** Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
- **Wiki audit cron job created:** Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
- **Ops profile Mnemosyne enabled:** Mnemosyne plugin enabled on ops profile with `profile_isolation: true` — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: `~/.hermes/profiles/ops/config.yaml` (`memory.mnemosyne.profile_isolation: true`, `allow_tool_override: false`).
- **OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04):** Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on `deepseek-flash-free` and `smart-route`/`good-free` across both instances. Reworked both combos: strategy `priority` → `fill-first`, dropped proven culprits (`238d5978/deepseek-v4-flash` + `-0731`, `nvidia` deepseek), healthy free models promoted, deepseek demoted to last resort. See [[incidents/2026-08-04-omniroute-free-deepseek-silent-stops]].
- **OmniRoute three-layer fix (2026-08-07):** npm package `.env` override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See [[systems/omniroute]].
- **Nanobot status degraded (2026-08-07):** CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See [[systems/nanobot]].
- **CT460 memory bumped (2026-08-07):** 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.
## Recent Changes (2026-08-08)
- **Cron job model repinning:** 3 jobs repinned from dead `freeapi/minimax-m2.7` → `smart-route` (IDs: `c8d52904e1d6`, `1b23c042e17d`, `d40085da631c`). Mosquito Tracker repinned from dead `freeapi/auto` → `smart-route` (ID: `1b31187397a5`). 9 stale-named cron jobs (e.g. `vertex/gemini-2.5-flash`, `gpt-5.6-terra`, `deepseek-v4-flash-free`) repinned to `smart-route`. Daily Marketing Job Radar swapped to `gpt-5.6-terra`.
- **Hermes UI vite config revert:** Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted `hermes-ui.service`. Production build now serves `/ui/` path correctly with relative asset paths. `hermes-desktop.martinwa.org` identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
- **Phase One estate review completed:** Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with `last_status=ok`, and CT333 Nanobot has a live `jobs.json` with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.
## Recent Changes (2026-08-09)
- **Portainer password reset (CT245):** Admin user is `anthony` (not `admin`). Password was forgotten; reset via `docker run --rm -v portainer_data:/data portainer/helper-reset-password` after stopping the Portainer container. Verified end-to-end with `POST /api/auth` → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on `:9443` (see [[systems/homepage-dashboard]]). Recovery procedure: stop container → run helper image → restart → verify with auth API.
- **SFTPGo security review (CT299):** Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path `//` resolves to `/` — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is `root`, (4) quota value `F1` on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.
## Recent Changes (2026-08-10)
- **Cron job failures — session storage issue:** Four cron jobs failed on 2026-08-10 with two distinct error patterns:
- **TimeoutError (idle for 601–602s):** "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
- **RuntimeError (session storage):** "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
- **Root cause:** Disk at 83% (11GB free on 65GB). `state.db` is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
- **Impact:** News digests and GitHub Trending did not deliver. Job Radar did not run.
- **See:** [[systems/scheduled-tasks]] Known Issues section.
- **OpenCode WebSocket proxy errors:** Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, `opencode serve`), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.
## Recent Changes (2026-08-14)
- **OmniRoute factory reset on CT601:** Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at `/opt/omniroute/factory-reset-backup-20260814-225524`. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See [[systems/omniroute]].
- **Hermes default model changed:** `auto/best-free` → `resilient-free` in `~/.hermes/config.yaml`. `auto/best-free` was misconfigured (no combo definition, routed to paid `gpt-5.6-sol`).
- **Multiple free combos broken by upstream model ID changes:** OpenRouter Poolside `laguna-s-2-1:free` → `laguna-s-2.1:free` (dash→dot); Groq `llama-4-scout-17b-16e-instruct` retired. Affected: `resilient-free`, `free-IA`, `free-stack`, `good-free`, `deepseek-flash-free`, `mimo-free`.
- **Guanaco router updated:** CT205 `ollama` container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See [[systems/guanaco]].
- **GPT-5.6 Luna research:** Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See [[systems/model-providers]].
- **Hermes UI restart:** Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
- **LiteLLM routing issue identified:** Hermes was previously pointed at LiteLLM (`http://litellm:4000/v1`, model `free-auto`) causing 6–26s per call. OmniRoute `auto/best-free` tested at 265ms. Config now points at OmniRoute.
## Recent Changes (2026-08-13)
- **OmniRoute migrated CT600 → CT601:** Provider renamed `custom:omniroute` → `custom:omni`; base URL `http://omniroute:20128/v1` → `http://omni:20128/v1`. Host `omni.kangaroo-eel.ts.net` (100.93.204.4, active) replaces `omniroute` (100.88.81.19, offline). CT600 stopped, CT601 running. DB path `/var/lib/omniroute/storage.sqlite` → `/opt/omniroute/storage.sqlite`; API key unchanged. All config (14) + cron-job (22) `custom:omniroute` refs repointed to `custom:omni`. Backup: `~/.hermes/config.yaml.bak.omniroute-removal`. See [[systems/omniroute]].
- **Evening stand-down briefing weather bugfix:** Weather collector returns a plain string (`+16°C Sunny`), but the three dashboard widgets (`evening.html`, `evening-grid.html`, `evening-briefing.html`) read object fields → `undefined`. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.
## Related
- [[systems/browser-backend]]
- [[systems/mnemosyne]]
- [[systems/scheduled-tasks]]
- [[comparisons/steel-vs-camoufox]]
- [[decisions/rollback-steel-to-camofox]]