Drift found and corrected against live sources (not guessed): - OmniRoute: CT600 omniroute is RUNNING and serving :20128 (combo API live); CT601 omni is STOPPED with onboot=0 (peer offline 54d). Wiki had these inverted. Hermes custom:omni resolves to omniroute -> CT600, not CT601. - Third OmniRoute node discovered: omniroute-1 at 100.78.63.20:20128, live (401), registered as custom provider omni-101 (5384 models, default smart-route, carries skills_hub/free-stack). Not in pct list; identity unconfirmed. - Ignis: TWO live instances. CT460 ignis (:8080, vault wiki/agent-estate-wiki) and CT200 ignis-ignis-1 (image ignis-server:latest, host 8092, vault /opt/ignis/vaults). Both return HTTP 200. Page now distinguishes them. - Tailscale: /agent-estate targets bare 127.0.0.1:8769 (was .../8769/agent-estate-role-dashboard.html); undocumented tailnet-only :9443 -> 127.0.0.1:5173 added; CT201 routes corrected (8085 targets + /steel 5173 + /steel-api 3000, replacing the wrong :8443 -> :8082 row). - Proxmox: rebuilt from live pct list + qm list. Added CT475 agentzero and VM110 omarchy (both previously missing); 282 ubuntu-server-xfce corrected to stopped; dead rows removed. VM403 open-claw and VM401 hermes-old absent from qm list - flagged unconfirmed, not deleted. - Nanobot: gateway nanobot-gateway (PID 2259048) alive on 8765/8766. It is NOT systemd nanobot.service/gateway.service (both return inactive - wrong unit names). Port 8900 no longer listening; "Serve API" claim flagged unverified. - Docker: old "runtime not available" was a CT460 socket artifact (root:docker srw-rw----, hermes not in group docker, no passwordless sudo in cron). Documented CT200 (~50 containers), CT333 (9 containers), CT460 via sudo. - Mattermost CT255 confirmed stopped; matches Tailscale peer offline 21d. Flagged, not investigated (no root cause): - mycompose-db-1 on CT200 is in Restarting (1) - crash-looping. 62 pages remain >21d stale and were NOT rewritten this pass (concepts, decisions, comparisons, most runbooks) - not probed against a live source, so reported not faked. Quartz rebuild step still inert (no quartz.config.ts) - public/ not refreshed. Artifacts: systems/skills-index.md regenerated, raw/stocktake/freshness-20261011.txt.
44 KiB
44 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Current State | concept | active | 2026-07-22 | 2026-10-11 | 2026-10-11 | 2026-10-25 | high |
|
|
Current State
Updated: 2026-10-11
Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST)
- OmniRoute CT600/CT601 roles are INVERTED. CT600
omniroute(100.88.81.19) is running and serving:20128(combo API returns live combos); CT601omni(100.93.204.4) is stopped withonboot=0(Tailscale peer offline 54d). Hermes resolvescustom:omni→omniroute:20128→ CT600. → systems/omniroute - A third OmniRoute node exists:
omniroute-1at100.78.63.20:20128, live (401 = auth-required), registered in Hermes config as provideromni-101(5384 models, defaultsmart-route). It carriesskills_hub(modelfree-stack) — the same provider family implicated in the 2026-10-09 timeout storm. Not present inpct list; identity unconfirmed. → systems/omniroute - Two Ignis instances exist, on different hosts. CT460
ignis(port 8080, vault/home/hermes/wiki/agent-estate-wiki, serving 200) and CT200ignis-ignis-1(imageignis-server:latest, host port 8092, vault/opt/ignis/vaults, serving 200). Both healthy; they are separate deployments. → systems/ignis - Tailscale route drift fixed:
/agent-estatetargets barehttp://127.0.0.1:8769(page had.../agent-estate-role-dashboard.html); a tailnet-only:9443→127.0.0.1:5173Serve was undocumented. CT201's live routes are 8085-based plus/steel(5173) and/steel-api(3000) — the old:8443→localhost:8082Homepage row is wrong. → infrastructure/tailscale - Proxmox inventory rebuilt from
pct list+qm list: CT475agentzeroand VM110omarchywere missing entirely; 282ubuntu-server-xfcecorrected to stopped; dead rows removed (310, 500, 104, 106, 101). VM403open-clawand VM401hermes-oldare absent fromqm list— flagged unconfirmed, not deleted. → infrastructure/proxmox - Nanobot verified on CT333:
nanobot-gateway(PID 2259048) alive on 8765/8766. It is not a systemdnanobot.service/gateway.service(both probes returninactive— wrong unit names); it runs as thenanobotuser service plus the gateway process. Port 8900 is no longer listening — previous "Serve API" claim flagged unverified. → agents/nanobot - Docker services page corrected: the old "Docker runtime not available" line was a CT460 socket artifact (
/var/run/docker.sockisroot:docker srw-rw----;hermesuser not in groupdocker; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → infrastructure/docker-services - Mattermost CT255 confirmed stopped, consistent with its Tailscale peer being offline 21d. Root cause of the unreachable endpoint still undetermined.
- ⚠
mycompose-db-1on CT200 is inRestarting (1)— crash-looping. Flagged; root cause NOT investigated this pass. - Stocktake artifacts: skills-index regenerated (346 skills / 70 categories),
raw/stocktake/freshness-20261011.txt= 62 stale / 24 fresh / 7 no-date of 93 pages. - Quartz rebuild still inert (no
quartz.config.ts) —public/was not refreshed. Carried open item; unchanged from prior days. - 62 pages remain >21d stale and were not corrected this pass: they were not probed against a live source, so they stay flagged rather than rewritten.
Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST)
The manual 2026-10-10 stocktake entry follows beneath this one.
- The cron fleet is clean again: 174
completed, 0failedon 2026-10-10 (AWST day,executions.db) - after 10-071/ 10-082/ 10-0915failures. 7 of the 10 storm incidents auto-closed during the day; 5 remain open over 3 jobs (1b23c042e17d+dca8482e4f76timeout/alertedsince 2026-10-09;0e753e9720fd+fad5a86979d4agent). Non-cleanlast_statusdown from 10 to 5. Job count 30 -> 29 (26 enabled / 3 disabled). -> systems/scheduled-tasks - The DHL monitor
62619aa6bcb3is retired - open item #1 closed. Anthony instructed removal again on the evening of 2026-10-10; the job is absent from livejobs.json(verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID. - The 2026-09-28 runtime-dependency class recurred - in the gateway env this time. The gateway's
activate_dependencies()loads one committed uv env (795020dbf24343cd9c52df24723dca83); it was missingmnemosyne/mnemosyne_hermesplus native deps, andelevenlabswas missing from the live managed env. Repaired 21:26-21:34, verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools)->activated; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - open. -> incidents/2026-09-28-hermes-managed-runtime-deps - New page: systems/paseo - Paseo coding-agent daemon, user unit
paseo-daemon.service, v0.11.2,0.0.0.0:6767, running since 2026-10-10 14:47:28. Pitfalls recorded:daemon runvsdaemon start,/usr/bin/paseois 0.7.2 while the daemon is 0.11.2, and a truncatedclinebinary presenting as a provider fault. - Mattermost unreachable from CT460 - configured endpoint
100.98.116.14:8065gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> systems/messaging-integrations - Health-monitor follow-ups (2026-10-10): "capture silent" was a false alarm (cron-only window);
check_embedding_coverage.pynow carries mode0755; 13 orphanmemory_embeddingsrows (oldest 2026-08-22) remain unverified. -> systems/mnemosyne
Recent Changes (2026-10-10)
- The 2026-10-09 timeout storm is over — the fleet is green again. Window tally (
executions.db, 2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163completed, 15failed. Every failure wasRuntimeError: Request timed out.on the routing aliases behindcustom:omni-101, confined to 00:00 → 15:00; all runs from 18:00 onward completed. 10 open incidents remain (ecc1fc_58b6c38296f3,08eb86_634f1a296f78,1b23c0_c35703f637b8,85edf6_18bedbeec5cd,dca848_23fcd2b18aaa,1a8e90_dc6b2d40ec44,1a8e90_789224ad589c,0e753e_e7f8194a7d06,fad5a8_c80b59a907f0,fad5a8_a13bbc837e1c). → incidents/2026-10-09-cron-provider-timeout-storm 1a8e90ea2271(stocktake) is running right now, not failed. Its 2026-10-09 03:00 run timed out before the generator produced output (so nofreshness-20261009.txtexists — the two-day artifact gap stands); the 2026-10-10 03:00:17 run isrunningat audit time.failure_streak: 2until it completes. This stocktake (run manually by this audit) regeneratedsystems/skills-index.md(346 → 346 skills / 70 categories, frontmatter dates refreshed) and wroteraw/stocktake/freshness-20261010.txt— 62 stale, 23 fresh, 7 no-date of 92 pages. See systems/scheduled-tasks.- Skills count reconciled: 347
SKILL.mdon disk vs 346 reported — unchanged off-by-one (the generator de-dups the nestedproductivity/obsidian-git-sync/obsidian-git-sync/duplicate). Carried open item. - 8 jobs carry a stale
last_status: error—0e753e9720fd(streak 0),fad5a86979d4(streak 0),dca8482e4f76(streak 1),1b23c042e17d(streak 1),85edf6772bca(streak 1),08eb86f02d37(streak 1),ecc1fc13655a(streak 1),1a8e90ea2271(streak 2). Two are pre-repin stale fields (streak 0), not live failures. Job count unchanged: 30 (27 enabled / 3 disabled). - DHL monitor
62619aa6bcb3still enabled — re-probed 2026-10-10:enabled: True,paused_at: null, schedule0 8,11,14,17,20 * * *. Anthony instructed its deletion on 2026-10-09 21:24 and a session reported it deleted. The parcel (8052838382) wasDelivered2026-10-07, so the monitor is redundant. This audit did not mutate cron state — retirement is open item #1 in systems/scheduled-tasks. - Quartz rebuild step confirmed still inert (no
quartz.config.ts,npxpresent but unreachable forquartz) — thepublic/static site was not refreshed by this stocktake. Skills-index and freshness artifacts regenerated and committed correctly; only the published-HTML step is a no-op. Carried open item from 2026-10-07. - Memory recall ranking fix still not persisted — the 2026-10-09 in-session "config saved" claim is not in live config; carried open item. Capture/vector health unchanged (760/760 working, 1,043/1,047 episodic embedded).
infrastructure/tailscale.mdre-verified 2026-10-10 —tailscale serve statusmatches the page exactly: 13 tailnet Serve routes (/,/ui,/wiki,/ignis,/webui,/gitea,/jobs,/desktop,/vnc-camofox,/synergy-prep,/agent-estate,/evening-grid.html,/ignis/:8443funnel/webhook) + the:8443/webhookpublic Funnel. No drift; page remains fresh (19d).
Recent Changes (2026-10-09)
- Cron fleet: the worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all
RuntimeError: Request timed out.on the aliases behindcustom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open;1a8e90ea2271sits atfailure_streak: 2. → incidents/2026-10-09-cron-provider-timeout-storm - The 2026-10-08 audit window was lost to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → log
62619aa6bcb3(DHL Tracking — Parcel 8052838382) is still enabled even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in systems/scheduled-tasks.- Memory recall relevance incident — a stale conversational episode outranked current-state facts; capture and vector health verified (760/760 working, 1,043/1,047 episodic), ranking fix not persisted in config. → incidents/2026-10-09-memory-recall-relevance
- Mnemosyne live counts (2026-10-09 22:27): 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → systems/mnemosyne
Recent Changes (2026-10-08)
- Cron fleet green for a third consecutive day — 0 failing. Job count unchanged at 30 (27 enabled / 3 disabled) and all 30
failure_streakcounters still read 0. Window tally (executions.db, 2026-10-07 00:00 → 2026-10-08 00:00 AWST): 176completed, 1failed, 1running(the nightly audit itself). 13 sessions started in the window, allsource = cron— no interactive (non-cron) sessions; 12 deliveries, 0 delivery failures. - First detect → notify → auto-resolve incident cycle since the managed-runtime event (DHL, 2026-10-07 08:00 → 11:01).
62619aa6bcb3(DHL Tracking, 5×/day) failed its 08:00 run insideensure_browser()withTimeoutError: timed out— Camofox briefly unresponsive on a browser HTTP call (dhl_tracking_monitor.py:233). The failure notice went out at 08:00:58 and incident62619a_f485ccfd4a0d(failure_type: timeout) auto-resolved at 11:01:13 when the next 3-hourly run completed; the 14:00, 17:00 and 20:00 runs also succeeded. No action needed — and the incident machinery behaved exactly as designed, which it had not been tested on since 2026-09-30. See systems/scheduled-tasks. - The tracked parcel arrived — the DHL monitor is now redundant. The 14:00 run on 2026-10-07 reported waybill
8052838382asDelivered.62619aa6bcb3still runs 5×/day for no remaining benefit; flagged as a removal candidate rather than removed. - The stocktake job proved its repin for a second day.
1a8e90ea2271completed automatically at 2026-10-07 03:00 — 346 skills / 70 categories (unchanged),raw/stocktake/freshness-20261007.txt(62 stale, 21 fresh, 7 no-date of 90 pages), committed to Gitea as8c63ef2. Two script-level findings came out of the run: (1) the Quartz rebuild step is a no-op — noquartz.config.tsis present andnpxis unavailable, so the published site is never refreshed; (2)skills-indexunder-counts by one — 347SKILL.mdfiles exist on disk while the page reports 346, because the generator de-dups the nestedproductivity/obsidian-git-sync/obsidian-git-sync/duplicate away. Both are now recorded as open issues in systems/scheduled-tasks. - Nothing else changed in the window. The three stale pre-repin
last_status: errorjobs (1b23c042e17d,fad5a86979d4,0e753e9720fd) remain — due Fri 08:00 / Mon 03:00 / 1st 09:00. Thecustom:omnistale-key references and the three bare-telegramtargets are unchanged. No new incident page was required, and no page was removed or archived.
Recent Changes (2026-10-07)
- Cron fleet green for a second consecutive day — 0 failing. Job count unchanged at 30 (27 enabled / 3 disabled) and all 30
failure_streakcounters read 0. Window tally (executions.db, 2026-10-06 00:00 → 2026-10-07 00:00 AWST): 176completed, 0failed, 1running(the nightly audit itself) — 177 rows. 14 sessions started in the window, allsource = cron; no interactive (non-cron) sessions. No new incident was opened. The repin wrap-up closures at 00:10–00:12 on 2026-10-06 fall inside this window but were already recorded by the 2026-10-06 audit. - The stocktake job proved its repin — first fully automatic run since 2026-09-25.
1a8e90ea2271(wiki-stocktake-maintenance, 03:00 daily) ran on schedule 2026-10-06 03:00:31 → 03:03:32 on its newcustom:omni-101/free-stackpin and completed cleanly. Its four HTTP-410 incidents (6df957a05ec6,d284168d3663,cda67de5b3e3,e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. It regeneratedsystems/skills-index.md(344 → 346 skills), wroteraw/stocktake/freshness-20261006.txtand committed to Gitea as33f4e95. Its stale pre-repinlast_status: erroris now cleared — livelast_status: ok, streak 0. The daily manualwiki-maintain.shrun is no longer needed. See systems/scheduled-tasks. - Three jobs still carry a stale pre-repin
last_status: error—1b23c042e17d,fad5a86979d4,0e753e9720fd(down from four). Each clears on its next scheduled slot (Fri 08:00 / Mon 03:00 / 1st 09:00); none is a live failure. The bare-telegramtarget warning still stands on066bb5191280and0e753e9720fd(unchanged). - Wiki vault mirror refreshed.
systems/skills-index.mdin/home/hermes/wiki/agent-estate-wiki/had drifted to 2026-10-03 / 344 skills while the canonical copy regenerated to 2026-10-06 / 346; this audit re-mirrored it along with the pages below. The two-wiki-tree drift remains a standing open item.
Recent Changes (2026-10-06)
- Dead model pins cleared across the whole fleet — failing set 7 → 0. All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream:
cliprox/terra(HTTP 401 connection auth, 3 jobs),ollama-cloud/qwen3.5:397b(HTTP 410, retired 2026-09-25),nvidia/deepseek-ai/deepseek-v4-flash(HTTP 400, dropped from the live catalog, 2 jobs) andcommand-code/gpt-5.6-terra(HTTP 403MODEL_NOT_IN_PLAN). They were moved tocustom:omni-101on routing aliases —free-stack×6,smart-route×1 — because the fault was raw-pin brittleness, not any one model. Everyfailure_streakinjobs.jsonnow reads 0. Three were verified by live run:79f9bb41ecf5(23:39),06a0a1c8704f(23:45) anddca8482e4f76(23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stalelast_status: erroruntil their next scheduled slot. See incidents/2026-10-05-dead-model-pin-repin. - The
cliprox/terraconnection incident is closed for cron purposes. No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See incidents/2026-10-01-cliproxy-connection-auth-expired. - GitHub Trending Radar's revoked-token diagnosis is refuted.
c0a9fa314542ran on schedule at 2026-10-05 09:00 and completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run. NogetMe→ 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired. - A stale failure notice, not a live outage, opened the review. Anthony acted on a
85edf6772bcanotice that had been sitting in topic346since the 2026-09-29 07:00 run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Checkexecutions.started_atand thecron_incidentsstate before acting on a notice. - Window tally (
executions.db, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185completed, 3failed(all pre-repin, 03:00–07:00), 2unknown(aborted first attempts of79f9bb41ecf5/dca8482e4f76, superseded by their successful re-runs) and 1running(the nightly audit itself). No interactive sessions other than the repin session.
Recent Changes (2026-10-05)
- Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7. Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is
79f9bb41ecf5(Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identicalcliprox/terraerror the incident predicted:HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. That is the last untested job on the pin, so the provider-connection auth expiry is now 3 of 3 confirmed — settled as a credential fault, not per-job. No review was produced;delivery_outcome: deliveredagain because the failure notice is the deliverable. See incidents/2026-10-01-cliproxy-connection-auth-expired. 1a8e90ea2271failed again — 2026-10-04 03:00 (HTTP 410). Thewiki-stocktake-maintenancejob remains pinned to the Ollama-retiredqwen3.5:397b; no change, still needs a repin. Its script is healthy. See systems/scheduled-tasks.- No interactive (non-cron) sessions in the window — all 10 sessions active in the past 24h were cron. The only durable changes are cron-fleet status, reflected above.
- Streak-number caveat for future audits: the
N failures in a rowfigure inhermes cron listis thefailure_streakcounter fromjobs.json, not a DB-derived count.~/.hermes/cron/executions.dbnow retains only rows from 2026-09-30 onward (1,001 rows), so DB recounts undercount older streaks. Live counters 2026-10-05:1a8e90ea2271×11,dca8482e4f76×7,fad5a86979d4×3. Quote the live counter, and note the retention floor.
Recent Changes (2026-10-04)
- Both Hermes instances updated on 2026-10-03, from opposite directions. CT460 (production) was rebuilt to
v0.21.5+6169.gab86a20(upstreamab86a20a, tree mtime 10:11 AWST) and CT450 (chaos) came up fromv0.19.0tov0.21.5+6170.ga510d6c(upstreama510d6c6, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reportsUpdate available: 3 commits behind. CT450 has no SSH — it is reached viassh root@100.96.100.82+pct exec 450, andhermesis not on the defaultpct execPATH. See incidents/2026-10-03-ct450-hermes-update. - Vector recall had been silently running keyword-only — found and fixed 2026-10-03.
fastembedwas missing from every uv-managed runtime, so the dense leg ofmnemosyne_recallcontributed no candidates and recall fell back to FTS5 without erroring. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime,FASTEMBED_CACHE_PATHpinned, andhermes-runtime-deps-guard.shextended to assert fastembed + cache path + model dir every 15 min. Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%), 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See incidents/2026-10-03-mnemosyne-vector-recall-fts-only and systems/mnemosyne. - Managed runtimes pruned 10 → 3. Only 3 uv-managed venvs remain under
~/.hermes/installs/019d0d114d22b7e6/environments/(two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale. - Cron fleet re-verified 2026-10-04 00:12 AWST: 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window.
79f9bb41ecf5— the last untestedcliprox/terrajob — is due 2026-10-04 18:00 AWST. See systems/scheduled-tasks. - Mattermost is unreachable from CT460 right now (
mattermost connect timed out after 30s, next retry in 300s, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed. - Wiki maintenance gotcha worth remembering:
mnemosyne doctorruns under the legacy Python 3.11venv/, wheresqlite_vecis not importable, so it printsvec0: present_but_unloadableand marks vector tiersunavailableeven when the gateway is embedding fine. Judge vector health withcheck_embedding_coverage.py, never with doctor'svec0line.
Recent Changes (2026-10-03)
- Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live
~/.hermes/cron/jobs.json). Failing set stays at 6, but the composition changed:1b23c042e17d(Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the identical HTTP 401 the cliprox incident predicted —HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. That makes 2 of the 3cliprox/terrajobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke.79f9bb41ecf5(Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due 2026-10-04 18:00. See incidents/2026-10-01-cliproxy-connection-auth-expired. - Streaks recounted from
executions.db— one number corrected:dca8482e4f766 → 7 (one more daily run), but1a8e90ea22718 → 7 — the ×8 carried since 2026-10-01 was an overcount; that job has exactly 7 recorded runs and all 7 failed. Recount before quoting a streak. - A delivered notice is not a successful job. Both
cliprox/terrafailures are recordeddelivery_outcome: deliveredbecause the failure notice is itself a deliverable — the executionstatusisfailedand no digest/plan was produced. Judge cron health on thestatuscolumn, not the delivery outcome. - Resolved —
1b23c042e17dstaledelivery_failedflag cleared: the 2026-09-30 retarget totelegram:-1004321904721:102worked exactly as predicted; its 2026-10-02 run delivered (then failed on the model pin). The prediction held. - systems/skills-index regenerated by hand: 341 → 344 skills. The daily stocktake job (
1a8e90ea2271) is still blocked on a retired model (HTTP 410,qwen3.5:397b), but its script is healthy —wiki-maintain.shwas run manually on 2026-10-03 and completed cleanly, refreshing the index and writingraw/stocktake/freshness-20261003.txt. One new skill appeared:fleet-wide-failure-triage(devops). Only the scheduled wrapper is broken; every non-model step of the stocktake now needs a manual run each day until it is repinned. - Work mailbox blocker documented — and it is older than it looks. A new incident page records that the
workOutlook account has been unreachable since 2026-08-26, not this week: the account uses password auth while the tenant advertises onlyXOAuth2, sohimalaya -a workdies withlogin not supported. The TCP/TLS path is fine — it is a tenant policy wall, not a bad password, and re-running the credential helper can never fix it. It needs an OAuth2 path. No cron job polls that mailbox, so this is an on-demand capability gap. See incidents/2026-08-26-work-outlook-basic-auth-disabled. - GitHub Trending Radar token claim downgraded to "unverified": the revoked-bot-token (
getMe→ 401) diagnosis could not be reproduced —c0a9fa314542's last recorded run (2026-09-30 19:22, off-schedule) completed and delivered. Re-confirm at the next scheduled run (Mon 2026-10-05 09:00) before acting on it.
Recent Changes (2026-10-02)
- Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live
~/.hermes/cron/jobs.json). Failing set 3 → 6: two new failures landed on 2026-10-01 —06a0a1c8704f(AI Newsletter Curation for Patrick, Thu 10:00) —HTTP 401 … All 1 connection(s) authentication expired — please reconnect in the dashboardon providercustom:Omniroute:20128, modelcliprox/terra. Its failure notice reached topic 346; no digest was produced. Not self-recovering — needs the OmniRoute connection re-authenticated. Two other enabled jobs share that pin (1b23c042e17d,79f9bb41ecf5) and will fail next run.0e753e9720fd(Facebook Birthday Monthly Scrape, 09:00) —HTTP 400,nvidia/deepseek-ai/deepseek-v4-flashabsent from the active NVIDIA catalog, confirming the earlier diagnosis that this is a shared-pin problem (siblingfad5a86979d4uses the same model).066bb5191280(Uber Eats Monthly Spend Report, 02:11) — report generated correctly; delivery failed withconfigured thread_id 34795 for telegram:1793951355 was not found.
- Carried-over failures, streaks advanced:
dca8482e4f76403MODEL_NOT_IN_PLAN→ streak 6;1a8e90ea2271HTTP 410 (retiredqwen3.5:397b) → streak 8;fad5a86979d4HTTP 400 → streak 3. All three still need a model repin. - Telegram delivery semantics corrected — a "failed" delivery is not always a lost message. Against
~/.hermes/cron/executions.db, the 2026-10-01 failures carry the fallback variant:configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id. Content reaches the DM, yet the run is recordeddelivery_outcome: failed. The 2026-09-30 incident'sMessage thread not foundvariant is the one that truly loses the message. Both variants are now documented in systems/messaging-integrations. - The wiki has two trees, and they have drifted in both directions (new finding, 2026-10-02): the canonical source is
/home/hermes/workspace/agent-estate-wiki— it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build (serve.py,package.json,public/) and is whatwiki-maintain.shcommits./home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault copy (pages nested underwiki/, no git remote) that the iPad reads. The nightly audit is told to write to the vault path, so edits that skip the canonical tree never reach Gitea or the published site. Worse, the trees have diverged backwards too:systems/ignis.md,systems/nextcloud.mdand threeincidents/pages exist only in the vault, whileraw/stocktake/*,incidents/2026-09-02-sftpgo-outage.md,reports/communities-wa-social-media-audit-2026.mdandagents/exist only in canonical.index.mdin canonical links[[systems/ignis]]and[[systems/nextcloud]], both of which are broken there. Any one-way sync will silently delete pages — reconcile before trusting either tree. See systems/scheduled-tasks. - Skills inventory: 344
SKILL.mdunder~/.hermes/skills/(live count 2026-10-02), against 341 in the last generated systems/skills-index (2026-09-27). The page cannot refresh while the stocktake job stays broken.
Recent Changes (2026-10-01)
- Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live
~/.hermes/cron/jobs.json). Failing set 4 → 3:cbfe434a3b5ereturned tookas predicted on 2026-09-30. The residual1a8e90ea2271dispatch error also cleared — its current failure is a different, upstream problem (below). - Telegram delivery failures root-caused and fixed: the target, not the topic, was wrong — a bare
deliver: "telegram"resolves to the home DM (1793951355), where group topic IDs cannot exist, so sends hard-failed and the message was lost.1b23c042e17dandc8d52904e1d6retargeted totelegram:-1004321904721:102and verified. Three jobs still carry baretelegramtargets (0e753e9720fd,fad5a86979d4,066bb5191280). See systems/messaging-integrations. - Three jobs pinned to models that no longer exist upstream — cannot self-recover:
1a8e90ea2271→qwen3.5:397bretired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running);dca8482e4f76→gpt-5.6-terraplan-gated (403MODEL_NOT_IN_PLAN, streak 5);fad5a86979d4→nvidia/…/deepseek-v4-flashabsent from the live catalog (HTTP 400, streak 3). Repin to thesmart-routealias. See systems/model-providers. - GitHub Trending Radar — revoked Telegram bot token:
c0a9fa314542builds its digest then fails the push (getMe→ 401). Token in/home/hermes/workspace/github-trending-radar/config.jsonneeds replacing from @BotFather. - Memory hygiene pass (2026-09-30): 36 gateway background-process notifications had been written into Mnemosyne as
[USER] [Anthony Martin]memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical factshermes_cron_model_defaultand the job-radar model corrected. See systems/mnemosyne. - Provider stack rebuilt from live config: 5 custom providers (
omni-101,Omniroute:20128,litellm,command-code,guacano). The provider keycustom:omnino longer exists; two cron jobs (e530c89f2be0,d40085da631c) still name it but have not failed, because the defaultcustomblock resolves to the same base URL. - Vision routing — request recorded, not applied: Anthony asked on 2026-09-30 that vision calls use the local Ollama at
http://anthonys-mac-mini:11434(gemma4). Live config still points atnous / stepfun/step-3.7-flash:freeas at 2026-10-01.
Recent Changes (2026-09-30)
- Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST: Jobs failed with
Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause: the uv-managed runtime venvs (10 under~/.hermes/installs/019d0d114d22b7e6/environments/*/venv) were missingdotenvandmnemosyne, so the worker died on import before acknowledging the job (ModuleNotFoundError: No module named 'dotenv'— 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed byhermes-runtime-deps-guard.shv2 (checks/repairs all managed venvs viapip install --target), run every 15 min by job96eb95ce12f2.weekly-save-restricted-bot-restartscript restored 19:17; that job isokagain. See incidents/2026-09-28-hermes-managed-runtime-deps. - Hermes upgraded since last audit: v0.19.0 (2026.7.20) → v0.21.5+3837.gbfda74c (2026.9.24), upstream
27062c34, update available. Skills 330 → 341. - Provider routing drift: default provider is
custom→http://100.78.63.20:20128(OmniRoute) with default modelsmart-route; auxiliary curatoropenai-codex / gpt-5.6-terra. Verified live 2026-09-30. - Open — Daily Marketing Job Radar blocked: job
dca8482e4f76pinned tocustom:omni-101/gpt-5.6-terra; upstreamcommand-codereturns403 MODEL_NOT_IN_PLAN. 5 consecutive failures (2026-09-26 → 30). See runbooks/job-radar. - Open — Telegram thread delivery failures (root cause found 2026-10-01):
telegram:1793951355thread34795"not found" for jobsfad5a86979d4and1b23c042e17d;c8d52904e1d6reports "Message thread not found". Cause was the delivery target, not the topic — a baretelegramresolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above. - Wiki tree structure confirmed (drift unresolved):
/home/hermes/workspace/agent-estate-wikiis the Gitea/Quartz source (this repo);/home/hermes/wiki/agent-estate-wikiis the Obsidian/Ignis vault (pages underwiki/, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree. - Multi-bot group chat question raised: Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.
Recent Changes (2026-09-02)
- SFTPGo outage resolved (02:18 AWST): Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's
tailscaledhad died, breaking tailnet reachability; PVE'spct startthen failed because 5 stalelxc-attachprobes from earlier diagnostics pinned CT299's cgroup (/sys/fs/cgroup/lxc/299/ns/.lxc). Killed those PIDs (3037951/3334928/3352203/3402511/4004148),pct stop 299 --skiplock+pct start 299, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end:tailscale ping sftpgo→ direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP bannerSSH-2.0-SFTPGo_2.7.0. See incidents/2026-09-02-sftpgo-outage.
Attested facts
- Config version:
_config_version: 46— Verified,~/.hermes/config.yaml, 2026-09-30 (previously recorded as 33 on 2026-07-25) - Default profile:
default— Verified, WebUI prompt, 2026-07-22 - Timezone:
Australia/Perth— User-provided + Verified in config, 2026-07-22 - Main provider:
custom→http://100.78.63.20:20128(OmniRoute) — Verified,~/.hermes/config.yaml, 2026-09-30 (previouslycustom:litellm→litellm:4000, thencustom:omni→omni.kangaroo-eel.ts.net) - Default model:
smart-route— Verified,~/.hermes/config.yaml, 2026-09-30 (previouslyfreellmapi-auto) - Fallback providers:
opencode-go / mimo-v2.5,opencode-zen / deepseek-v4-flash-free,nous / stepfun/step-3.7-flash:free,ollama-cloud / qwen3.5:397b,kilocode / poolside/laguna-m.1:free,custom / litellm:4000— Verified, config, 2026-09-30 - Auxiliary curator model:
openai-codex / gpt-5.6-terra— Verified, config, 2026-09-30 - Memory backend: Mnemosyne — Verified, config, 2026-07-22
- Memory enabled: true — Verified, config, 2026-07-22
- Streaming enabled: true — Verified, config, 2026-07-22
- Dashboard public URL:
https://hermes.martinwa.org/— Verified, config, 2026-07-22 - Pet config: enabled/slug null — Verified, config, 2026-07-22
- WebUI connected platforms: local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22
Recent Changes (2026-08-02)
- Browser backend reverted: Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation.
browser.cloud_provider: camofox,camera_provider: camofox. Steel plugin disabled. See comparisons/steel-vs-camoufox and decisions/rollback-steel-to-camofox. - Mnemosyne capture fix (2026-08-01): Automatic conversational capture was broken by stringified YAML (
sync_roles: "['user']"). Fixed by rewriting structured Mnemosyne settings with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure. - Mnemosyne vector coverage gap discovered: Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
- DeepSeek V4 Flash 0731: Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (
^[a-zA-Z0-9_-]+$) — may cause issues with OmniRoute-transformed tool names. - Uber Eats report upgraded: Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with
camofox-browser-automation+browser-backend-verificationskills. - GitHub Trending Radar — Daily Digest removed: Stopped by user on 2026-08-02. Weekly roundup still active.
- Ops profile renamed: Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
- Wiki audit cron job created: Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
- Ops profile Mnemosyne enabled: Mnemosyne plugin enabled on ops profile with
profile_isolation: true— Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config:~/.hermes/profiles/ops/config.yaml(memory.mnemosyne.profile_isolation: true,allow_tool_override: false). - OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04): Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on
deepseek-flash-freeandsmart-route/good-freeacross both instances. Reworked both combos: strategypriority→fill-first, dropped proven culprits (238d5978/deepseek-v4-flash+-0731,nvidiadeepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops. - OmniRoute three-layer fix (2026-08-07): npm package
.envoverride (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See systems/omniroute. - Nanobot status degraded (2026-08-07): CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See systems/nanobot.
- CT460 memory bumped (2026-08-07): 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.
Recent Changes (2026-08-08)
- Cron job model repinning: 3 jobs repinned from dead
freeapi/minimax-m2.7→smart-route(IDs:c8d52904e1d6,1b23c042e17d,d40085da631c). Mosquito Tracker repinned from deadfreeapi/auto→smart-route(ID:1b31187397a5). 9 stale-named cron jobs (e.g.vertex/gemini-2.5-flash,gpt-5.6-terra,deepseek-v4-flash-free) repinned tosmart-route. Daily Marketing Job Radar swapped togpt-5.6-terra. - Hermes UI vite config revert: Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted
hermes-ui.service. Production build now serves/ui/path correctly with relative asset paths.hermes-desktop.martinwa.orgidentified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460. - Phase One estate review completed: Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with
last_status=ok, and CT333 Nanobot has a livejobs.jsonwith 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.
Recent Changes (2026-08-09)
- Portainer password reset (CT245): Admin user is
anthony(notadmin). Password was forgotten; reset viadocker run --rm -v portainer_data:/data portainer/helper-reset-passwordafter stopping the Portainer container. Verified end-to-end withPOST /api/auth→ HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on:9443(see systems/homepage-dashboard). Recovery procedure: stop container → run helper image → restart → verify with auth API. - SFTPGo security review (CT299): Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path
//resolves to/— full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user isroot, (4) quota valueF1on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.
Recent Changes (2026-08-10)
- Cron job failures — session storage issue: Four cron jobs failed on 2026-08-10 with two distinct error patterns:
- TimeoutError (idle for 601–602s): "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
- RuntimeError (session storage): "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
- Root cause: Disk at 83% (11GB free on 65GB).
state.dbis 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint. - Impact: News digests and GitHub Trending did not deliver. Job Radar did not run.
- See: systems/scheduled-tasks Known Issues section.
- OpenCode WebSocket proxy errors: Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564,
opencode serve), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.
Recent Changes (2026-08-14)
- OmniRoute factory reset on CT601: Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at
/opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See systems/omniroute. - Hermes default model changed:
auto/best-free→resilient-freein~/.hermes/config.yaml.auto/best-freewas misconfigured (no combo definition, routed to paidgpt-5.6-sol). - Multiple free combos broken by upstream model ID changes: OpenRouter Poolside
laguna-s-2-1:free→laguna-s-2.1:free(dash→dot); Groqllama-4-scout-17b-16e-instructretired. Affected:resilient-free,free-IA,free-stack,good-free,deepseek-flash-free,mimo-free. - Guanaco router updated: CT205
ollamacontainer — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See systems/guanaco. - GPT-5.6 Luna research: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See systems/model-providers.
- Hermes UI restart: Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
- LiteLLM routing issue identified: Hermes was previously pointed at LiteLLM (
http://litellm:4000/v1, modelfree-auto) causing 6–26s per call. OmniRouteauto/best-freetested at 265ms. Config now points at OmniRoute.
Recent Changes (2026-08-13)
- OmniRoute migrated CT600 → CT601: Provider renamed
custom:omniroute→custom:omni; base URLhttp://omniroute:20128/v1→http://omni:20128/v1. Hostomni.kangaroo-eel.ts.net(100.93.204.4, active) replacesomniroute(100.88.81.19, offline). CT600 stopped, CT601 running. DB path/var/lib/omniroute/storage.sqlite→/opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22)custom:omnirouterefs repointed tocustom:omni. Backup:~/.hermes/config.yaml.bak.omniroute-removal. See systems/omniroute. - Evening stand-down briefing weather bugfix: Weather collector returns a plain string (
+16°C Sunny), but the three dashboard widgets (evening.html,evening-grid.html,evening-briefing.html) read object fields →undefined. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.