Files
agent-estate-wiki/current-state.md
T
Tony0410 e033a08f17 audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)
- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
2026-11-13 00:41:15 +08:00

136 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Current State concept active 2026-07-22 2026-11-13 2026-11-13 2026-11-13 high
system
live-cron-list
live-sqlite-cron-executions
live-sqlite-sessions
live-probe
live-executions-db
live-cron-incidents
live-deliveries-db
live-sessions-db

Current State

Updated: 2026-11-13

Recent Changes (2026-11-12, nightly audit; live probes 2026-11-12 23:59 – 2026-11-13 00:4x AWST)

Clean day. The estate is incident-clean, and the only open item is an upgrade that cannot be installed.

  • The last open incident closed itself. cbfe43_ed01845310d8 (the 2026-11-10 audit's 429, incidents/2026-11-10-wiki-audit-429-all-targets-skipped) went closed_at: 2026-11-12T00:25:36.133387+08:00, state: resolved, when the next slot of the same job ran clean — cbfe434a3b5e 2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s, completed, delivery carrying both windows. cron_incidents: 0 unclosed of 137. No configuration produced this; the pool's reset at ≈ 06:36:57 on 11-11 did, and the 07:0x cohort confirmed it.
  • State reads: fleet 29 (26 enabled / 3 disabled) — unchanged for a seventh window. Ledger — 2026-11-11: 172 rows, 172 completed / 0 failed; 2026-11-12: 176 rows, 175 completed / 0 failed / 1 running (this audit); 2026-11-13 so far: 7 / 7 completed. Deliveries 11-11: 7 / 7 delivered; 11-12: 9 / 9 delivered, 0 for_failure. Executions retention floor 2026-11-07T00:00:42, max 2026-11-13T00:30:55, 1,001 rows in the table.
  • New open item — a refused upgrade, not a failure. The attempt to take hermes-hq-push past its installed 0.9.42 was blocked by the pre-install plugin security scan: "Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec))", and "--force does not override a dangerous verdict." The critical is the plugin's own skills/dashboard-tile/SKILL.md one-liner (echo '<json>' | python3 … = obfuscation). Verified in source: there is no per-verdict override anywhere — plugins.scan_on_install: false disables scanning for every future install, and the plugin guard hardcodes trust_level="community" (the skills-side allowlist machinery is not read by it). Installed copy untouched, enabled, still delivering. Fix = PR #2 on mrcharlesiv/hermes-hq-plugins (author Tony0410, fix/tile-skill-heredoc, OPEN / MERGEABLE, mergedAt: null, +7/−2 in one file), pushed from a fork because gh is read-only upstream. → incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan
  • Config untouched and consistent. Live config.yaml is byte-identical to the 2026-11-12 18:14:52 snapshot (565,433 B). The only drift since 11-09 is the model catalogue block (lines 1916-1941): 6 models removed, 9 added. No unrelated settings moved.
  • The "stored hold-and-ask instruction" is not stored. The alleged standing rule about plugin security-scan warnings was hunted exhaustively — a recursive sweep of ~/.hermes (184,716 files, generated/venv/browser-profile trees excluded) found zero matches in any editable text file; the sentence exists only in state.db conversation history (58 rows). Nothing was removed, nothing was reverted — there was no persisted policy to undo. Procedure recorded as the skill stored-instruction-search-and-removal; installed skills 299 → 300.
  • The recurring pre-dispatch-filter signature was re-confirmed on a second job and a second pin by the 2026-11-12 03:00 stocktake — still no failover for the free-tier cohort (incidents/2026-10-21-omniroute-pre-dispatch-filter-skip).

Recent Changes (2026-11-11, nightly audit; live probes 2026-11-11 23:59 – 2026-11-12 00:1x AWST)

Recovery night — plus one failure, and it was the auditor's own. → incidents/2026-11-10-wiki-audit-429-all-targets-skipped

  • The 2026-11-10 audit died on its own 429. cbfe434a3b5e ran 2026-11-10T23:59:54.486572 → 2026-11-11T00:12:29.646987 = 12 m 35 s, failed, delivery suppressed: RuntimeError: HTTP 429: Service temporarily unavailable: all targets were skipped by pre-dispatch filters (credits 100% conn:3021bab6) reset after 6h 24m 31s — 6 candidates, attempted 0, tenant tn-5ud77abtl1 quota exceeded. Row cbfe43_ed01845310d8 (rate_limit) was raised at 00:12:28 but alerted_at: NULL — it was the estate's only open incident row; it has since closed resolved at 2026-11-12T00:25:36 (0 of 137 unclosed). Pool reset ≈ 06:36:57; the 07:0x cohort ran clean. ⇒ The 2026-11-10 window was never audited. This audit covers two windows.
  • State reads: fleet 29 (26 enabled / 3 disabled); non-clean last_status 2 (was 1) — 066bb5191280 Uber Eats (delivery_failed, streak 0) and cbfe434a3b5e (error, streak 1). Ledger — 2026-11-09: 174 completed / 0 failed; 2026-11-10: 157 rows, 155 completed / 2 failed; 2026-11-11: 172 rows, 171 completed / 1 running / 0 failed (15 distinct jobs) — first zero-failure day since 11-09. Deliveries 11-11: 7 rows, 7 / 7 delivered, 0 for_failure (last at 18:45:58).
  • Stocktake recovered with output. 1a8e90ea2271 03:00:56.081504 → 04:11:38.724852 = 1 h 10 m 42 s, completed, 36 stale pages re-verified → 0 stale, commits 341dda9 + c108eb1, freshness report raw/stocktake/freshness-20261111.txt (6,235 B). 341dda9 also rewrote the live fallback chain in runbooks/test-openrouter (primary back to smart-route / custom:omni-101), corrected the CT333 clawtest naming claim, and replaced the nous refresh-token reading with the 404-credits signature. Closes incidents/2026-11-09-stocktake-completes-with-zero-edits. ⚠ The job is still pinned to free-stack, the combo recorded broken on 2026-08-14, and succeeded.
  • Job Radar recovered with output. Stage 1 dca8482e4f76 07:01:08.358004 → 07:37:34.076224 = 36 m 26 s, completed, writing radar-ref/handoff/2026-11-11.json (22,682 B, mtime 07:59) — the first handoff since 2026-11-06. Stage 2 5fc292ff0f3b: recovery fire 07:35:56.632032 → 08:03:47.816811 (delivered), scheduled 11:30:56 → 11:35:18, 14:30:55 → 14:34:56 (delivered). Row 5fc292_773822455856 auto-closed resolved 2026-11-11T08:03:47.644226, closing incidents/2026-11-09-job-radar-stage1-stall-silent-noop.
  • Longest runs. 11-11: stocktake 1 h 10 m 42 s, Radar S1 36 m 26 s, Radar S2 27 m 51 s (recovery fire), Newsletter Digest ecc1fc13655a 12:01:02.571781 → 12:19:38.965387 (18 m 36 s), 85edf6772bca 07:01:08.287627 → 07:11:42.858722 (10 m 34 s). Nothing over 71 minutes. 11-10: Radar S2 scheduled slot 11:30:35.267155 → 22:18:49.814215 = 10 h 48 m 14 s, failed / interrupted, delivery delivered with for_failure: 1.
  • Provider state (unchanged config; worsening symptoms). config.yaml mtime still 2026-11-09 22:58:07 — nothing repaired. Top model custom:omni-101 / smart-route (http://100.78.63.20:20128). fallback_providers (6): nous meituan/longcat-2.5-preview:free → opencode-zen deepseek-v4.1-flash (EXPIRED key) → custom:guacano → custom:litellm kilo-reflection-beam-free → kilocode poolside/laguna-m.1:free → custom (http://litellm:4000/v1) or-inclusionai_ling-2_6-1t. The expired opencode-zen key is no longer latent — 7 × HTTP 401 api_key_expired on 11-11 (07:01:13 … 19:19:39), each immediately after nous returned 14 × 404 insufficient_credits_for_paid_model (07:01:11 … last 19:19:38, all from session 20261031_190813_95fbde, source desktop, opened 2026-10-31, still open, pinned to that paid model). Two consecutive dead hops on one chain. → systems/opencode-go
  • Auxiliary slots: nine slots plus free_only: true. Eight chain no-think/freeapi/deepseek-v4-flash → no-think/kg/kilo-auto/free on custom:omni-101. pulse_memory remains the sole exception with no fallback_chain ({"provider": "nous", "model": "anthropic/claude-haiku-4.5"}) — 19 aux log lines on 11-11 across 06:0x / 08:0x / 13:0x / 18:0x, including 11 × "main provider custom:omni-101 is unavailable and no fallback_chain / fallback_providers is configured — refusing to guess another logged-in provider". → incidents/2026-10-22-nous-portal-refresh-token-invalid
  • Error log: ~/.hermes/logs/errors.log = 6,179 lines through 2026-11-12 00:10:11.
  • Still open, untouched this window: Uber Eats Monthly Spend Report delivery_failed streak 0 (no execution row on 11-11 — monthly cadence); disabled trio unchanged (Important-mail monitor, Rhino AgentMail Inbox Watcher, AgentMail Rhino Inbox Poll); the three inherited items below.

Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)

  • ✅ 2026-11-09 was the first failed-free cron day since 2026-11-05: 174 rows — 173 completed / 0 failed / 1 running (this audit). Trend: 11-06 1 failed / 172, 11-07 3 / 163, 11-08 1 / 170. Deliveries 8/8 delivered, 0 for_failure.
  • ⚠️ The stocktake ran 9 h 10 m, closed completed, and landed nothing — third day running. 1a8e90ea2271 (03:00:43 → 12:10:47) delivered suppressed; its own output file says RUN INCOMPLETE, tool-iteration budget exhausted during verification, zero edits and no commit. failure_streak 3 → 0 and last_status ok are the harness's verdict on the process, not the work. The job is still pinned to custom:omni-101 / free-stack — a combo recorded broken on 2026-08-14. → incidents/2026-11-09-stocktake-completes-with-zero-edits, systems/scheduled-tasks
  • ⚠️ The Job Radar landed nothing on 11-09. Stage 1 (dca8482e4f76) ran 07:00:48 → 16:10:12 = 9 h 09 m 24 s and never wrote radar-ref/handoff/2026-11-09.json (newest handoff on disk is 2026-11-06 — nothing has landed since). Both Stage-2 slots (5fc292ff0f3b) fired before Stage 1 finished (11:30:38 and 14:30:34) and declined silently. The run's own last words: "I stopped retrying because I kept running read_file 5 times without making progress." Same silent family as 11-02/11-03, new trigger (read loop, not iteration cap), and again no incident row was raised. → incidents/2026-11-09-job-radar-stage1-stall-silent-noop
  • ✅ The 11-08 pre_tool_call hook cascade did not recur. errors.log (1,035 retained lines) contains no pre_tool_call timeout entry at all; its only two hook lines are benign skips (on_session_end 2026-11-09 22:58:35, pre_llm_call 2026-11-10 00:02:54). No 11-09 run outside the two documented long-runners exceeded 74 min. → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade
  • Unclosed incident rows 3 → 0 (cron_incidents: 134 rows, 0 open). The stocktake cluster closed automatically at 2026-11-09T12:10:47.125205 — the instant the overrunning run finished; none of the three ever alerted. → systems/scheduled-tasks
  • Tailscale: 58 non-funnel nodes (57 peers + hermes), 42 online. octop is online again (dark since LastSeen 2026-11-08T13:13:04Z); 8 offline ages re-drifted — cloudreve 106d → 108d, elementsynapse 109d → 111d, freshrss-new 82d → 84d, conduit 62d → 65d, octoagent 6d → 8d, omarchybook 17d → 19d, openmausbot 17d → 19d, ubuntu-server-xfce 78d → 81d; the 6 ephemeral codex-environment-* peers' last-seen stamps were corrected. → infrastructure/tailscale
  • OmniRoute omniroute-1 (CT101, 100.78.63.20): /v1/combos → HTTP 200, 55 combos, free-stack present — one ahead of the page's last reading (54). /v1/models on both nodes and CT600's /v1/combos return HTTP 401 (alive, auth required), so the 6,480-model catalog figure dated 2026-10-26 could not be re-verified this window and was not changed. → systems/omniroute
  • Not verified this window (gap recorded, no page changed): CT450/CT460 Hermes version strings — pct exec 450 -- sh -lc 'hermes --version' resolves the known-stale root-level /usr/local/bin/hermes (v0.19.0), not the user-level install, and CT460's hermes is not on the PATH in that shell, so infrastructure/hosts's CT450/CT460 version lines stand as written; and CT450/CT460 docker Up days (lxc-attach … Failed to exec "docker").

Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)

  • ⚠ The conduit_push pre_tool_call hook was the day's fault, and it is fixed — plugins.hook_callback_timeout 30 → 120 s. The plugin is the only pre_tool_call registrant on the host, and its callback overran Hermes's 30 s hook budget 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03 (errors.log: 12 lines on 11-07, 46 on 11-08; plus 89 "skipped after previous timeout or while still running" suppression lines). Every tool call degraded to slow-then-suppressed. Fixed 2026-11-08 13:35 AWST; the last timeout line is still 08:21:03, re-verified at audit time. The plugin's caching was measured, not assumed, and was innocent — 3.1 s cold / 0.00 s warm (session_titles.py:21 CACHE_SECONDS = 30.0); the trigger is load on the 2-core CT460 (load average 5.89, state.db with a 40 MB WAL). → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade, systems/conduit
  • ⚠ state.db is now 1.89 GB (1,890,656,256 B; 2,901 sessions / 197,682 messages, live 40,157,672 B WAL) — up from 1.7 GB, and it is the load source behind the spike above. The 2026-08-11 corruption remnant state.db.corrupted.20260811_151052 (624.1 MB) was deleted in this window; state.db.recovered.recovery.json (14,566 B) remains, and mnemosyne.db.corrupted.20260809_120404 (23.6 MB) was not cleaned. → systems/hermes-agent, incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts
  • ✅ The Mnemosyne Health Monitor's no-agent conversion is verified end-to-end. The 18:00 slot (agent mode) ran 12:00:51 → 18:00:42 = 5 h 59 m 51 s; the converted job's first slot ran 2026-11-09 00:00:48 → 00:02:02 = 73 s, exit 0, silent, writing a 159 B output file against 105,982 B for the last agent-mode one. Live coverage working 1030/1030 = 100.0 %, episodic 1192/1192 = 100.0 %, so the < 95 % alert is correctly quiet. → systems/mnemosyne, systems/scheduled-tasks
  • ⚠ The stocktake failed a third consecutive day, on a new signature. 1a8e90ea2271 ran 2026-11-08 03:01:11 → 04:36:08 and died on RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text. — the previous two red days read "lost its durable fire claim ownership". failure_streak 3. Working hypothesis, not proven: the hook above holds a tool result open in exactly the way this error describes. Unclosed incident rows 2 → 3, and all three are silent (detected, alerted_at: null) — but the alert path works: the Newsletter Digest row 85edf6_0877ca1e0eac alerted 07:23:07 and resolved at 08:06:54. → systems/scheduled-tasks, incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain
  • 2026-11-08 settled: 169 rows — 168 completed / 1 failed / 1 running (this audit). Deliveries 8 / 8 delivered. Fleet 29 jobs (26 / 3) — a sixth consecutive audit at the same count, though updated_at moved to 2026-11-09T00:06:50+08:00 with no definition change. Nothing added, retired, disabled or repinned. → systems/scheduled-tasks

Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)

  • Hermes on CT460 is now 0.23.0 (was 0.22.5): hermes update, 2026-11-07 05:05:13 → 05:43:10, bf0203a794a07aac3ff764972ab9087baa4e4f4e → 2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19. ⚠ The snapshot stage was skipped again (updates.pre_update_backup mode off) → there is still no rollback point. Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55 (live pid 3109529). → systems/hermes-agent, agents/hermes-production
  • ⚠ The restart drain cost two cron runs their fire claims. A restart requested at ~04:29:52 sat in after-turn mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing Mnemosyne Health Monitor (ea79a2107e36, 00:00:42.918 → 05:02:22.870) and wiki-stocktake-maintenance (1a8e90ea2271, 03:00:31.515 → 05:02:22.174), both closed failed on lost its durable fire claim ownership. systemd restarted the unit at 05:01:00 (restart counter is at 3). → incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain
  • ⚠ Two incident rows are open and neither alerted. 1a8e90_628d914f1f97 (first seen 04:29:54.815) and 1a8e90_2f370ef764f5 (05:02:21.573) both sit detected for wiki-stocktake-maintenance with alerted_at: null — the red row, no alert shape first recorded for Job Radar on 11-06.
  • ⚠ The 03:00 stocktake slot has now failed two days running (failure_streak: 2, last_status: error, 2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not left dirty.
  • camofox is now an enabled plugin (v1.1.0), not a core feature — installed by the 0.23.0 update. No browser.* config key changed. → systems/browser-backend
  • conduit_push self-updated 0.122.2 → 0.134.4 (git source, tree HEAD ef3d2f2) and recorded its own last step restarting_gateway = failed — the gateway it wanted to restart is systemd-managed. → systems/conduit
  • hermes-hq-push reinstalled 0.9.27 → 0.9.42 (upstream d26850e7, 60/60 files) at 06:42; its web routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. → systems/hermes-hq-push
  • ⚠ The OpenCode Go credential is expired — HTTP 401, code: api_key_expired, on …/zen/go/v1/models (probed twice in-window). Configuration alive, credential dead; no 2026-11-07 execution failed on an auth error, so nothing was consuming it yet. → systems/opencode-go
  • ⚠ Skill inventory moved: 302 → 299 skills (68 → 67 categories). Four skills went into ~/.hermes/skills/.archive/ in one pass at 2026-11-07 07:43 — cron-model-repair, fleet-wide-failure-triage, hermes-instance-update, mail-lookup — and the parked archive grew 46 → 50. No active SKILL.md references any of them, so if this was a curator consolidation the "absorbed into X" pointer is missing. microsoft-office appeared in the same window (Productivity 25 → 26). → systems/skills-index, systems/scheduled-tasks
  • Cron day (2026-11-07): 163 rows — 159 completed / 3 failed / 1 running; deliveries 7/7 delivered, 0 for_failure. Fleet unchanged at 29 jobs / 26 enabled. → systems/scheduled-tasks
  • incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts remains OPEN — the re-check found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.

Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)

  • The tailnet lost a peer and the m93p guest list lost a guest — both corrections came from live probes, not the age scan. infrastructure/tailscale was rebuilt from tailscale status --json: octop (100.92.163.122) is back online (peer record Online: true, tailscale ping → pong from octop via DERP(syd) in 98ms; it remains no LXC or VM on either hypervisor — an orphaned tailnet peer), and the 9 remaining offline ages were refreshed (+1d each). All 51 live IPv4 peers match the table's IPs exactly, with no missing peers and no orphan rows (52 rows = 51 peers + hermes itself). The CT460 Serve route table still matches tailscale serve status exactly (12 paths on :443); both previously-flagged dead targets were re-confirmed dead — still no listener on 8769 (/agent-estate) or 9123 (/synergy-prep).
  • ⚠ m93p actually runs 20 LXCs, not 19 — CT107 (mem0) was missing from the Proxmox page. Live pct list on m93p returns 20 LXCs — 18 running, 2 stopped (CT998 nightbridge, CT1264 openmausbot). The omitted guest is CT107 mem0 (mem0-mcp.service active on :8890, pct exec 107 -- tailscale ip -4 → 100.73.94.53). Added to infrastructure/proxmox; the 2026-10-31 reading of 19 was already short for the same reason. pve still returns exactly 17 LXCs + 4 VMs, and both hosts still run the identical pve-manager 9.2.10 build.
  • ⚠ m93p's uptime claim was wrong. The page read "up 1 day, 12:02 … rebooted ~2026-10-30 16:0x". Live uptime -s shows the host booted 2026-11-05 21:19:52 (up 1 day, 7:18 at probe time, load 0.76 / 1.08 / 1.24 — normal). The 2026-10-30 reboot was real but belonged to pve (boot 2026-10-30 16:24:00), not m93p.
  • Storage percentages refreshed on both hosts against live pvesm status — pve's ssd0 moved 76.43 % → 78.29 % (~152 GiB free), fred-qnap 82.77 % → 84.21 %, local 79.66 % → 81.62 %; on m93p the iMac shares and local moved 60.25 % → 76.21 % and omb-internal 57.92 % → 64.66 %.
  • Docker inventories diffed set-wise against live docker ps on all three hosts. CT200 matches 61 running / 68 total exactly; CT201 matches 20 / 26 exactly; CT333 matches 8 / 9 — but its enumerated list still opened with flamboyant_mahavira, which is not running (live name is nostalgic_keldysh, created 2026-11-05 22:15:57). Corrected in infrastructure/docker-services.
  • Two small cross-page corrections. infrastructure/network-map's /gitea route was annotated "Gitea (ct207 tailnet IP)" — it is CT221 on m93p (100.82.100.128; ct207's own IP is 100.82.40.23); the target answers HTTP 200. agents/hermes-chaos's version line read "v0.22.3 … local 5b147ab036d6 — the same build as CT460", which contradicted itself: live hermes --version on CT450 is v0.22.5 … bf0203a794a0, now genuinely identical to CT460's. agents/hermes-production's claim that 2 cron jobs still pin custom:omni is no longer true — live jobs.json shows 0 jobs on the bare key and 21 on custom:omni-101.
  • Cron fleet re-verified unchanged: 29 jobs (26 enabled / 3 disabled), matching the nightly audit hours earlier. CT450's fleet is 25 jobs (13 enabled / 12 disabled) with 9 shared names — confirmed by a live set-diff of both jobs.json files. → systems/scheduled-tasks

Recent Changes (2026-11-07, nightly audit; live probes 2026-11-07 00:3x-00:5x AWST)

  • A red row with no alert: the Job Radar's 11:30 slot was discarded as stale while a concurrent fire held the claim. 2026-11-06 settled 172 rows across 16 jobs - 170 completed / 1 failed / 1 running (this audit). The failure is 5fc292ff0f3b's 11:30:32 slot, interrupted at 13:26:38 and closed 13:26:44 on Fire claim ownership lost; stale result was discarded. - because a direct fire of the same job (08:27:09 -> 13:26:22, 4 h 59 m 13 s) was still holding the claim and delivered first. Every previous occurrence of this string came from a restart orphaning a run; this one had no restart. The failed row raised no cron_incidents row (newest ledger activity is still the 2026-11-05T12:36:17 closures), so nothing alerted. The day still landed complete via the 14:30 slot (12 m 27 s, MISSING_COVER_LETTER_COUNT 0) -> incidents/2026-11-06-job-radar-concurrent-fire-claim-discard.
  • The stocktake's overrun is fully unwound. 1a8e90ea2271 ran 03:00:32 -> 04:11:35 (1 h 11 m 03 s) against 7 h 40 m / 5 h 41 m / 3 h 02 m on 11-03/11-04/11-05, and committed normally.
  • Stage 1 ran clean for a second day (dca8482e4f76, 07:00:44 -> 08:29:21, 1 h 28 m 37 s, completed, suppressed), and the 11-04 Open entry about its two lost candidate sets is cleared in substance - its 11-03 ledger row closed resolved at 2026-11-05T12:36:17 and both runs since have been clean.
  • Ledger clean, deliveries clean: unclosed incident rows 0 -> 0; deliveries 10 rows / 10 delivered / 0 for_failure; fleet 29 jobs (26 enabled / 3 disabled) - fifth consecutive audit at the same count, non-clean last_status still 1 (066bb5191280, delivery_failed, streak 0) -> systems/scheduled-tasks.
  • The previous audit's open question closed itself: the Mnemosyne Health Monitor's 11-05 12:00:45 slot (still in flight 12 h+ with no delivery at the last audit) completed at 2026-11-06T00:14:39 after 12 h 13 m 54 s and delivered; its 11-06 12:00 slot then took 7 h 47 m 56 s and delivered too. The one still-open page is unchanged: incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.

Recent Changes (2026-11-06, nightly audit; live probes 2026-11-06 00:1x-00:4x AWST)

  • A clean ledger day and an empty incident ledger. 2026-11-05 settled 175 rows - 173 completed / 2 running / 0 failed across 18 jobs: the first zero-failed day since 2026-11-02. Unclosed incident rows 3 -> 0 - the Tech & AI Newsletter Digest row closed at 07:07:42 and both Job Radar - Discovery rows at 12:36:17, when that job finally completed after 5 h 35 m 35 s. Fleet non-clean last_status 3 -> 1 (only 066bb5191280, delivery_failed). Deliveries 10 rows / 10 delivered -> systems/scheduled-tasks.
  • The stocktake's three-day overrun streak broke, and it committed. 1a8e90ea2271 ran 03:00:32 -> 06:02:30 (3 h 01 m 58 s) - 7 h 40 m -> 5 h 41 m -> 3 h 02 m over three days - and committed 2f3fc99 at 05:55, against its 11-04 state of "reported incomplete, committed nothing".
  • Long runs, not failures: the Mnemosyne Health Monitor's 00:00 slot took 7 h 01 m 28 s (normal 17-31 m) and its 12:00:45 slot was still running 12 h later with no delivery; Stage 2's 14:30 slot took 8 h 56 m 17 s and delivered at 23:27:09.
  • New open incident - live secrets stored verbatim. Three .env values (TELEGRAM_BOT_TOKEN, MATTERMOST_TOKEN, HERMES_CUSTOM_OMNIROUTE_20128_API_KEY) sit in plaintext in Mnemosyne rows - including the FTS mirrors) - 4 + 6 + 7 = 17 rows - reproduced in full by each of the 7 uncompressed rotation snapshots; all four values also sit verbatim in the 1.7 GB session store state.db, the likely capture origin. CAMOFOX_API_KEY sits verbatim in the prompts of two enabled Facebook Birthday jobs (0e753e9720fd, fad5a86979d4) and hence in cron/deliveries.db. Found, not remediated; no value is recorded anywhere -> incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts.
  • Fleet unchanged: 29 jobs (26 enabled / 3 disabled), a fourth audit at the same count; no job added, retired, disabled or repinned.

Recent Changes (2026-11-05, stocktake; live probes 2026-11-05 05:1x-05:3x AWST)

  • The systems/skills-index.md flip-flop is FIXED at the root - it had two writers, not one bad writer. The repo wiki-maintain.sh (302 skills / 68 categories) and ~/.hermes/scripts/wiki-maintain.sh (mtime 2026-08-17, 349 skills / 26 categories - it used Path.rglob() and so counted the parked .archive/ tree) are different scripts with different generators, and wiki-maintenance/SKILL.md line 569 pointed the 23:59 nightly audit at the stale one while the 03:00 stocktake ran the repo one. Each job overwrote the other's page; neither raised an error. The stale copy is now a delegating wrapper with no generator logic, and the repo script gained --no-build so the wrapper honours the old interface without a second generator. Verified by running it -> 302 / 68. -> incidents/2026-11-05-skills-index-generator-divergence
  • Docker inventory corrected against live probes on all five hosts: CT200 61 running / 68 total; CT333's first container renamed again (nostalgic_leavitt -> flamboyant_mahavira); CT201 20/26, CT460 1/1, CT1001 1/9. ⚠ Corrected 2026-11-06 (stocktake): the "the seeder does not exist on any host" claim in this entry was WRONG - it was a point-in-time sampling artefact. The worldmonitor-seeder-* containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer every 4h -> worldmonitor-seeder-scheduled; worldmonitor-seed-fast.timer every 15min -> worldmonitor-seeder-fast), so a docker ps taken between runs sees none of them. Live at 2026-11-06 03:1x both were present; live at 2026-11-05 05:1x neither was. "Absent" must never be read as "removed." CT333's container has also been renamed again: flamboyant_mahavira -> nostalgic_keldysh (created 2026-11-05T22:15:57Z). -> infrastructure/docker-services
  • Tailscale node table re-verified programmatically: 52 table rows vs 52 live IPv4 peers - no live peer missing, no orphan row, every online/offline state matches. Two offline ages corrected (conduit 60d, ubuntu-server-xfce 76d). Serve table still matches tailscale serve status exactly (12 paths on :443 + the :8443 funnel + the :9443 route). ⚠ Corrected 2026-11-06 (stocktake): the "every online/offline state matches" claim in this entry no longer holds - anthonys-mac-mini has come back online (peer record Online: true, tailscale ping -> pong) and 8 offline ages have drifted by +1d. Two Serve routes (/agent-estate -> 8769, /synergy-prep -> 9123) are dead - no listener - while still present in tailscale serve status. -> infrastructure/tailscale
  • Freshness now 0 stale (was 8 on 11-04) - all 8 re-verified and refreshed. Skills count corrected to 302 / 68 categories on infrastructure/hosts and index.

Recent Changes (2026-11-05, nightly audit; live probes 2026-11-05 00:0x-00:5x AWST)

  • A dead route took out two 07:00 jobs at once - and the safety-net slot saved the day. dca8482e4f76 (Daily Marketing Job Radar - Discovery) and 85edf6772bca (Tech & AI Newsletter Digest (Analytical)) both died on RuntimeError: Upstream stream failed before completion. - 36 seconds apart, at 09:41:44 and 09:42:20, each after 2 h 41 m on custom:omni-101 / smart-route. Neither produced output; both raised and alerted a cron_incidents row. The 11:30 Stage 2 slot then reconstructed the missing handoff from Stage 1's candidate files, re-verified every candidate live in Camofox, and wrote 5 roles (board 479 -> 484) + 1 canonical cover letter (MISSING_COVER_LETTER_COUNT 0) before consuming the handoff at 16:08:59 - see incidents/2026-11-04-upstream-stream-failure-0700-cohort.
  • Fleet: 29 jobs (26 enabled / 3 disabled). In-window ledger: 170 rows - 167 completed / 2 failed / 1 running. (fleet and last row corrected 2026-11-05 01:3x - the first draft read 158 jobs / 144 / 14 and a closing item_suppressed row; live jobs.json + hermes cron list give 29 / 26 / 3, and the 170th 11-04 row is this audit, running. 11-03 has since settled to 171 completed / 1 failed.) Non-clean last_status 2 -> 3; unclosed incident rows 1 -> 3 (all state = alerted, incl. the 11-03 timeout row dca848_dfd4bea84f2a). Deliveries on 11-04: 8 rows, 8 delivered - the failure notices were delivered, not silent.
  • The cover-letter route is only alive by override. regenerate_cover_letter.py line 82 reads CL_MODEL, defaulting to deepseek-paid-smart, which returned HTTP 402 during the 11-04 run; that process was pinned to gemini/gemini-3.5-flash. No CL_MODEL exists in ~/.hermes/.env, so the override did not persist. Endpoint check 2026-11-05: omniroute:20128 -> 401 and 100.78.63.20:20128 -> 401 (live, key-gated - that is auth, not health). Keyed re-probe 2026-11-05 01:33:12 AWST: a completion request with deepseek-paid-smart returns 503 ALL_TARGETS_SKIPPED (poolSize 5, attempted 0) on omniroute:20128, but 200 on 100.78.63.20:20128 (omniroute-1), which resolved the alias to deepseek/deepseek-v4-flash. So the model is undispatchable on the CT600 node only - an override or the omniroute-1 route is required.
  • CT299 froze a second time. A manual vzdump 299 ... --storage ssd0 started 2026-11-04 08:30:04 and wedged; the guest was frozen until ~00:30 on 11-05 (~16 h). Live probe 2026-11-05 00:54:21 AWST: LOCK: 0, cgroup.freeze=0, ns/cgroup.freeze=0, guest ALIVE, sftpgo: active, 0 vzdump processes; SFTPGo reachable from CT460 at 100.88.223.70:8080. Abandoned temp dir /mnt/ssd0/vzdump-tmp/vzdumptmp3750030_299 still on disk. -> incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage
  • wiki-stocktake-maintenance overran for the third day running - 1a8e90ea2271 ran 03:00:31 -> 08:42:17 (5 h 41 m 46 s), hit the turn ceiling, reported incomplete, delivery suppressed. Pin: custom:omni-101 / free-stack.

Recent Changes (2026-10-30, nightly audit; live probes 2026-10-30 00:0x AWST)

  • Conduit voice calls were repaired — after three faults in series — and the first two real calls in the product's history are now on record. Anthony asked why Conduit never answered when he called. In order: (A) voice.speech_server.url held the truncated wss://…:8443/v1/, which the plugin dials verbatim, so the Mac answered 403 and the plugin blamed the key; (B) dashboard.basic_auth.session_ttl_seconds was the 12-hour default, and an expired phone session gets WS 4401 with no Hermes turn and no error visible on the phone (the iPhone re-authenticated at 08:00:59, minutes after a silent call); (C) both host providers were dead — tts.provider: openai → Mac /v1/speech 502, stt.provider: groq → 403 / Cloudflare 1010. Fixed 08:02–08:4x: URL → …:8443/v1/realtime, TTL → 2 592 000 s (30 d), stt.provider: local (faster-whisper base), tts.provider: ollama_local (6.94 s of real speech verified); dashboard restarted 08:03:26. Verified end to end by driving the whole call path, then by calls at 08:38:04 and 08:49:36 — conduit-calls.json → history is no longer empty. → incidents/2026-10-29-conduit-voice-silence, systems/conduit, systems/hermes-agent
  • The Job Radar's two-stage split worked end to end for the first time. Stage 1 (dca8482e4f76, deliver: local) ran 07:00:59 → 10:16:12 (3 h 15 m 13 s), closed completed, and wrote ~/.hermes/radar-ref/handoff/2026-10-29.json — 170 discovered / 33 duplicates avoided / 37 verified, sources seek ok, linkedin ok, indeed blocked, wa_gov ok. Stage 2 (5fc292ff0f3b) consumed the handoff at 10:40:59, closed 10:44:48 completed and delivered to topic 1030 in the same second: 20 pages / 20 page_ids / 2 cover letters / verified_cover_letter_count 2 / missing_cover_letter_count 0, Notion board 455 → 475. Both of stage 1's incident rows (dca848_e93a538ade6b, dca848_0a2f03b5952b) closed resolved at 10:16:12; the 11:30 and 14:30 slots then no-oped as designed (10 m 46 s / 1 m 51 s). → incidents/2026-10-27-job-radar-budget-exhausted, incidents/2026-10-28-job-radar-stage1-iteration-cap, runbooks/job-radar
  • Cron: 2026-10-29 settled 177 completed / 0 failed — the first zero-failure day since 2026-10-24 — with 10 deliveries, all 10 delivered, 0 failures. Non-clean last_status 5 → 3; unclosed incident rows 4 → 1 (the two radar rows closed resolved at 10:16:12). Fleet unchanged at 29 jobs (26 enabled / 3 disabled) at that audit.
  • Cron: 2026-10-30 closed 138 / 0 — after a 5 h 30 m HOST OUTAGE — and the "auto-paused" health monitor was never dead, it was running. Fleet 29 jobs (26 enabled / 3 disabled) (jobs.json.updated_at 2026-10-31T00:29:59+08:00), re-verified 2026-10-31 00:3x. Zero execution rows exist between 10:58:20 and 16:37:21 — PVE pve rebooted (boot 2026-10-30 16:24:00), CT460 came back 16:28:00, the old gateway life (pid=3913841) was logged UNCLEANLY at 16:35:50 ("no exit path ran — SIGKILL / OOM / VM death" — a candidate list, not a diagnosis; root cause not established, no OOM evidence), and 8 catch-up runs fired at 16:37:21–16:38:22 → incidents/2026-10-30-ct460-host-outage-scheduler-gap. Against 10-29's 177 / 0, that is ≈39 lost slots and no failures. ea79a2107e36 (Mnemosyne Health Monitor) is ENABLED and clean again — last_status: ok, streak 0, incident ea79a2_d93c3acdc4f2 closed resolved 2026-10-30T10:02:48.662 when its 00:01:08 slot finished after 10 h 01 m 40 s; its 12:01 slot was lost to the outage and is the still-running 16:37:21 catch-up. Non-clean last_status 4 → 3, failing 1 → 0, unclosed incident rows 2 → 1 (survivor: 0e753e_e7f8194a7d06, due 11-01). deliveries.db finished 10-30 with 8 rows, all delivered (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is superseded.
  • ⚠ Superseded by the 2026-10-31 entry above — read that first. Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure notice was never delivered.** ea79a2107e36's 00:01:08 slot closed unknown at 01:04:22 on a gateway-connection failure ("Cron execution unverified; reconcile ~/.hermes/cron/admissions/ad897cbc9f…json: Gateway connection failed; no local fallback"), opening incident ea79a2_d93c3acdc4f2 (detected), and the scheduler auto-paused the job. Fleet is now 29 jobs (25 enabled / 4 disabled); non-clean last_status is 4 (2 error + 2 delivery_failed); ea79a2107e36 carries failure_streak: 1. ⚠ The failure notice itself failed to send — hermes cron list reports send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed, and deliveries.db holds zero rows for 2026-10-30, so the only trace is the incident row. Unclosed incident rows 1 → 2. Every other previously-red job (dca8482e4f76, 1b23c042e17d, ecc1fc13655a, 08eb86f02d37, 85edf6772bca, 0e519874d280, c0a9fa314542) has cleared to ok with streak 0 on its own slot. Pins: 18 of the 25 enabled jobs pin custom:omni-101 (11 smart-route, 6 free-stack, 1 deepseek-paid-smart); 7 enabled rows are no_agent jobs with no pin. → incidents/2026-10-30-mnemosyne-health-monitor-paused

Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 00:2x AWST)

  • The radar's silent budget overrun became a recorded failure — and the radar is now two jobs. Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 → 11:03:54 (4 h 2 m 46 s), closed failed, delivery delivered, and the delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery, verification, dedup and classification completed; Notion ingestion, cover-letter generation and the final cover-letter audit did not execute. The prior day (10-26) had the same overrun but closed completed, hiding it in a green row. Incident row dca848_e93a538ade6b opened 11:03:53, alerted 11:03:59, still open at this audit (failure_type: auth — misclassified). → new page incidents/2026-10-27-job-radar-budget-exhausted
  • Response: the radar was split into a two-stage pipeline the same day. Stage 1 dca8482e4f76 — Discovery (0 7 * * 1-5, deliver: local, failures → topic 1030) writes a handoff and triggers stage 2; new stage 2 5fc292ff0f3b — Daily Marketing Job Radar - Write and Letters (30 11,14 * * 1-5, Telegram topic 1030, created 2026-10-27T17:34:15) lands the Notion entries, cover letters and audit. Two test fires of stage 2 on 10-27 (17:35:00 → 17:36:52, 17:38:38 → 17:43:42) both completed with delivery suppressed; the first real 07:00 → 11:30 cycle had not yet run. Fleet 28 → 29 jobs (26 / 3). → runbooks/job-radar
  • The 2026-10-26 orphan class did not recur, and its two incident rows closed. ea79a2_a4661554a3c2 and ea79a2_c3ab2d784ba3 auto-closed 00:38:32 (unclosed rows 3 → 2, the 2 being the Facebook birthday monthly scrape from 10-01 and the new radar row). No 10-27 run besides the radar's failed, so the watchdog and every other job were clean all day.
  • Non-clean last_status 4 → 5 (churn): ea79a2107e36 cleared to ok; dca8482e4f76 joined as error (streak 1) and 08eb86f02d37 re-joined as delivery_queued. Still non-clean: 0e753e9720fd (error), fad5a86979d4 (delivery_failed), 066bb5191280 (delivery_failed). Pins unchanged.
  • Also updated: incidents/2026-10-27-job-radar-budget-exhausted.md (new), incidents/2026-10-26-planned-restart-orphaned-cron-run.md (recurrence check), runbooks/job-radar.md, systems/scheduled-tasks.md, systems/hermes-agent.md (TTS provider correction), systems/conduit.md (voice pipeline), infrastructure/tailscale.md (peer table re-verified), systems/skills-index.md (regenerated 10-27, committed), index.md, log.md. Nothing removed or archived. No secrets written.

Recent Changes (2026-10-28, nightly audit; live probes 2026-10-28 22:00 – 2026-10-29 00:3x AWST)

The radar's two-stage split ran for real, and landed nothing — stage 1 hit the agent iteration cap

dca8482e4f76 (stage 1, Discovery) ran 07:00:32 → 10:33:18 (3 h 32 m 46 s) and closed failed on "hit the agent iteration cap before the handoff could be written" — failure_streak: 2. Stage 1 never produced a handoff (~/.hermes/radar-ref/handoff/ holds only 2026-10-27.json), so the first real two-stage cycle delivered no Notion writes and no cover letters — the same outcome as 10-27, by a different route.

Both stage-2 slots then did exactly what they were built to do: 5fc292ff0f3b fired 11:30:27 → 11:31:31 and 14:30:27 → 14:31:43, both completed with delivery_outcome: suppressed — a stage-2 run with no handoff is a silent no-op, not a failure (the recovery rule in runbooks/job-radar), now proven in production. The two post-split failures also stopped at different ceilings: 10-27 at the tool-call budget (4 h 2 m, writes not started), 10-28 at the iteration cap (3 h 32 m, handoff not written) — less of the pipeline completed despite the shorter wall clock. Open radar incident rows: dca848_e93a538ade6b (10-27) and dca848_0a2f03b5952b (10-28), both open. Detail: incidents/2026-10-28-job-radar-stage1-iteration-cap.

The 03:00 stocktake failed on a provider 400 — and its incident row never alerted

1a8e90ea2271 ran 03:00:28 → 03:37:58 (37 m 30 s) and closed failed (streak 1, delivery suppressed) with RuntimeError: [400] Failed to deserialize the JSON body into the target type: duplicate field 'reasoning_content' at line 1 column 512203 against custom:omni-101 / free-stack — the response the provider was asked to deserialize carried a duplicated reasoning_content field. It performed its rebuild (302 skills → systems/skills-index.md mtime 03:01:40; freshness report raw/stocktake/freshness-20261028.txt → 2 stale / 103 fresh / 4 no-date of 109 pages) and then died before the commit, leaving the wiki tree dirty for the rest of the day. Its incident row 1a8e90_f57ab4cafab8 sits in state detected with no alert — the only outward sign was the job's own error status. The 2026-10-29 audit committed the two artefacts.

Steel left the stack: fully removed — Camofox is the sole backend

browser.cloud_provider and browser.camera_provider are back to camofox (engine: auto, cdp_url: "") in config.yaml (mtime 2026-10-29 00:15:13), with backups taken at every step (config.yaml.bak-pre-cloudprovider-fix-20261028T220057Z 22:00:57, and config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z written 00:13:24). The browser-steel plugin was removed from ~/.hermes/plugins/, and no browser-steel key remains in plugins.disabled or plugins.entries, so the plugin_steel toolset is no longer selectable. Steel is now fully REMOVED, not merely down: on CT201 the containers steel-browser-api-1 / steel-browser-ui-1, the steel-network Docker network, the images (steel-browser-api:fixed and both ghcr.io Steel images) and the project dir /root/steel-browser were deleted (~6 GB reclaimed; disk 70G used → 64G used), and the Tailscale Serve routes /steel and /steel-api were removed — CT201's remaining routes are only / → 127.0.0.1:8085 and /mgr → 127.0.0.1:8084 on both :443 and :8443, with nothing listening on 3000, 5173 or 9223. CT333's second, separate Steel copy (steel-browser, ghcr.io/steel-dev/steel-browser:latest, 5.53 GB, created 2026-05-30) was removed with its image. Hermes core contains zero Steel references, so there is no Steel fallback path in the tree — returning to Steel means redeploying it (the compose file and Dockerfile are archived on CT201 under /root/_archived-steel/; the images remain re-pullable from ghcr.io). ⚠ STEEL_BASE_URL is no longer in ~/.hermes/.env — it survived only in the inherited environment of processes already running at the cleanup, and the gateway was restarted on 2026-10-29 01:21 AWST to clear it (a couple of older, unrelated services still carry it) — no core Hermes code reads it, so it is a runtime-only leftover, not a config value. Camofox is the sole backend and healthy: REST http://100.118.5.51:9377 (and http://localhost:9377), /health → 200, /tabs → {"running":true,"tabs":[]}. Detail: systems/browser-backend, decisions/rollback-steel-to-camofox, comparisons/steel-vs-camoufox.

⚠ Superseded: an earlier 2026-10-29 00:3x probe described Steel only as "not reachable — endpoint dark, still running on CT201". That was already out of date — Steel is removed, not down.

Fleet and tallies

29 jobs (26 enabled / 3 disabled) — unchanged. Non-clean last_status: 5 (0e753e9720fd error, fad5a86979d4 delivery_failed, dca8482e4f76 error streak 2, 066bb5191280 delivery_failed, 1a8e90ea2271 error streak 1). Pins: 12 smart-route / 6 free-stack / 7 unpinned / 1 deepseek-paid-smart. 2026-10-28 settled 168 completed / 2 failed / 1 running; 2026-10-29 so far 6 completed / 1 running; 2026-10-27 settled 173 / 1 (revised up by one late run after the 2026-10-28 audit). No Hindsight service is running on this box (:8888 closed, no hindsight units) — the standing memory-backend correction still holds and needs no change.

Recent Changes (2026-10-27, nightly audit; live probes 2026-10-27 00:1x–00:5x AWST)

  • A planned gateway restart still orphaned a live cron run — and this time it alerted loudly. A planned SIGTERM at 2026-10-26 14:33:35 produced a shutdown drain that reported timed_out=True after just 1.35 s of its 30 s budget and interrupted ea79a2107e36 (Mnemosyne Health Monitor, 12:01 slot) 2 h 34 m in. It closed failed on a new string — lost its durable fire claim ownership — against the 2026-10-20 orphan's silent unknown, and raised two cron_incidents rows (14:33:37 unverified / reconcile admission, 14:35:59 fire claim) that are still alerted; the failure was delivered at 14:35:59. Two Mnemosyne Watchdog reds followed and self-resolved at 15:16:12. Window 2026-10-26 settled 169 / 3 and all three failures are this one fault — so the fleet's failure count is one interruption, not a systemic turn. Fleet unchanged at 28 jobs (25 enabled / 3 disabled); non-clean last_status 5 → 4; unclosed incident rows 2 → 3; pins unchanged. → new page incidents/2026-10-26-planned-restart-orphaned-cron-run
  • One incident row closed, and the 2026-10-16 503 burst is fully shut. c0a9fa_764558b5ae83 (GitHub Trending Radar) auto-closed 09:13:00 when its 09:00 slot ran clean — the last open row of the resource-pressure fault → incidents/2026-10-16-omniroute-503-resource-pressure. No execution row carries the 503 string on 10-25 or 10-26. The job stays monitoring: single-node exposure unchanged.
  • Two quiet degradations worth naming. fad5a86979d4's 03:08 slot completed but delivered delivery_failed — Thread 34795 not found (the gateway's own shutdown notice hit the same dead thread at 14:33:37); and the radar's 07:00 slot closed completed at 09:10 while delivering “RUN INCOMPLETE — tool-call budget exhausted” — no Notion writes, no cover letters, batch finished by hand. That morning the cover-letter PDF pipeline moved to Chromium over the canonical HTML with model deepseek-paid-smart and a 600 s dashboard timeout → runbooks/job-radar
  • Also updated: incidents/2026-10-26-planned-restart-orphaned-cron-run.md (new), incidents/2026-10-20-gateway-unclean-exit-orphaned-run.md (recurrence), incidents/2026-10-16-omniroute-503-resource-pressure.md, runbooks/job-radar.md, systems/scheduled-tasks.md, index.md, log.md. Nothing removed or archived.

Recent Changes (2026-10-25, nightly audit; live probes 2026-10-25 00:1x–00:5x AWST) (superseded by the 2026-10-27 block above)

  • A third consecutive clean cron day — and an off-cron fault that is not the fleet's. Window 2026-10-24: 167 completed / 0 failed (+ this audit running); 2026-10-23 settled at 170 / 0 (169 at the 10-24 audit). Fleet unchanged at 28 jobs (25 enabled / 3 disabled), 4 non-clean last_status, 2 unclosed incident rows — nothing opened, nothing closed in-window — pins unchanged. 14 sessions: 11 cron + 3 desktop. → systems/scheduled-tasks
  • ⚠ New: the agent refused to phone Anthony, three times. On 2026-10-24 12:53–12:56 he asked to be called to test Conduit calls (Call me now to test calls. → Yes. You do → CAL ME). All three replies asserted the agent has no calling capability — with zero tool calls in the session — while conduit_call_user was one tool_search away and his call settings were permissive. Capability asserted instead of discovered; the tool itself is fine. → new page incidents/2026-10-24-conduit-call-request-refused
  • Conduit, corrected and updated. Plugin 0.69.0 / HEAD f000bf0 (22 commits on) while the gateway process still runs the 10-23 code and the memory-timeout patch is uncommitted and now five releases behind HEAD. Device phone-status contact last 10-23 19:10:30; health ingestion to 10-24 09:59:59; store 569.9 MB / 1,520,229 events / 3,028 buckets / 124 windows (the 10-24 page's 308 / 62 was a narrower grouping). ⚠ conduit-calls.json → history is empty — the 10-22 memory-fix incident's “two calls closed cleanly 00:03:23 / 00:06:57” was a false positive (cron turn-ends, not calls), so that fix still has no positive call evidence. → systems/conduit, incidents/2026-10-22-conduit-memory-start-timeout
  • Also updated: systems/conduit.md, incidents/2026-10-22-conduit-memory-start-timeout.md (correction), systems/scheduled-tasks.md, index.md, log.md. Nothing removed or archived.

Recent Changes (2026-10-24, nightly audit; live probes 2026-10-24 00:1x–00:3x AWST) (superseded by the 2026-10-25 block above)

  • A clean cron day, and the 2026-10-09 timeout storm is finally closed. Window 2026-10-23: 169 completed / 0 failed (+ this audit running) — the third zero-failure day in four; 2026-10-22 settled at 173 / 0. 1b23c042e17d (Enhanced Weekend Adventure Planner, weekly Fri 08:00) ran clean on its 10-23 slot, closing the storm's last two incident rows (1b23c0_c35703f637b8, 1b23c0_103cbe04a79b) at 08:04 — all 12 rows closed, job ok again. Unclosed incident rows 4 → 2, non-clean last_status 5 → 4, fleet unchanged at 28 jobs (25 enabled / 3 disabled), pins unchanged. → incidents/2026-10-09-cron-provider-timeout-storm, systems/scheduled-tasks
  • New: Conduit is documented. The dashboard plugin + iOS app that puts Hermes on Anthony's phone (voice calls, cards, push) now has a system page — plugin tree, version 0.64.0 (HEAD eaa14be), upstream kaishi00/hermes-conduit-notifier, gateway on :9119, one registered device (iPhone, app 2.0.0), its state/DB files and their sizes, and the uncommitted patch it carries. → systems/conduit
  • ⚠ New fault, found and fixed in-window: Conduit voice calls were being abandoned by the memory layer. The plugin hard-coded a 10 s budget for one request including the Mnemosyne provider start, while Mnemosyne's cold start for a conduit-voice session runs 10–45 s. Three calls hit it (10-22 11:28:17, 10-22 13:16:37, 10-23 18:29:16: Memory provider 'mnemosyne' start for Conduit timed out; abandoning the call) and ran without recall. Patched in the plugin's working tree to a 60 s default (CONDUIT_MEMORY_TIMEOUT_S override) plus a TokenError guard that proceeds without recall instead of aborting; the dashboard restarted 18:54:43 and no abort line exists after it, with two calls closing cleanly at 00:03:23 and 00:06:57 on 10-24. The patch is uncommitted — a plugin update reverts it. → incidents/2026-10-22-conduit-memory-start-timeout, systems/mnemosyne
  • Sessions: 14 in the window — 13 cron + 1 desktop (10-23 18:26:12, the phone-call session above); no other interactive sessions.
  • Stocktake: the 10-22 sixfold overrun did not repeat — the 10-23 03:07 slot ran and committed (250c735: freshness 2 stale / 94 fresh, skills-index 306). → systems/skills-index
  • Housekeeping: frontmatter added to incidents/2026-07-29-mnemosyne-relay.md and incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md, both of which sat outside the SCHEMA (rule 4). Freshness: 102 pages; 3 carry a verified_on older than 21 days — infrastructure/hosts.md, runbooks/provider-health.md, runbooks/update-hermes-safely.md (all 94 d). Nothing removed or archived.

Recent Changes (2026-10-23, nightly audit; live probes 2026-10-23 00:0x–00:16 AWST) (superseded by the 2026-10-24 entry above)

  • First zero-failure cron day in three days, and five incidents cleared at once. Window 2026-10-22: 172 completed / 0 failed (+ this audit running); 4 runs had already completed on 10-23 by 00:0x. Five incident rows closed and none opened — 0e5198_be2291809e56 + 0e5198_ec2fac98a91d (08:01:40), 06a0a1_9e0ff4eedc49 (10:14:34), c8d529_e8c70250f9be (18:17:59), dca848_014affeb6954 (07:59:22) — so unclosed rows fall 9 → 4 (3 alerted / 1 detected). Non-clean last_status falls 9 → 5: 1b23c042e17d (error, streak 2, next slot Fri 2026-10-23 08:00), c0a9fa314542 (error, 1, Mon 10-26), 0e753e9720fd (error, 0, 11-01), fad5a86979d4 + 066bb5191280 (delivery_failed, Mon 10-26 / 11-01). All four cleared job rows had been red since 2026-10-15 (the omni-101 503 burst) and each ran clean on its first slot of 10-22. → incidents/2026-10-16-omniroute-503-resource-pressure
  • Correction to the 2026-10-21 tally. A DB recount by both started_at and claimed_at (AWST) puts 2026-10-21 at 168 completed / 1 failed, three below the 171 / 1 recorded on 10-22. Retention does not explain it (floor 2026-10-08); the delta is recorded rather than overwritten. → systems/scheduled-tasks
  • ⚠ The stocktake ran six times long, reported completed, and committed nothing. 1a8e90ea2271 ran 2026-10-22 03:00:28 → 04:38:19 (97 m 58 s) against 16 m 01 s on 10-21, delivery suppressed; its regenerated systems/skills-index.md (mtime 03:00) was still uncommitted at audit time and its diagnosed fixes were never applied. A completed cron row is not evidence the work happened. → incidents/2026-08-15-skills-index-drift
  • Skills index fixed at the root tonight. The page counted 41 parked skills under ~/.hermes/skills/.archive/, and two scripts generated it with different shapes (repo wiki-maintain.sh, find-based → 347 with a ## .Archive section; stale ~/.hermes/scripts/wiki-maintain.sh, rglob-based → 348 without). The repo script now excludes */.archive/*, prints the archived count as an exclusion, and names hermes skills list as the authoritative loadable figure. Page regenerated: 306 skills across 69 categories (was 347). Live counts: filesystem 348, with symlinks 376, minus archive (41) + apple/ (4) = 331, loader 329 enabled (34 hub / 52 builtin / 243 local). → systems/skills-index
  • Wiki maintenance deployed. Quartz parsed 102 files, emitted and deployed 602 files, and hermes-wiki-serve.service restarted 00:16 AWST. Freshness: 2 stale / 94 fresh / 6 no-date of 102 pages — reports/communities-wa-social-media-audit-2026.md (56 d) plus, newly over the threshold, systems/model-providers.md (22 d, updated 2026-10-01; needs a pass against the live OmniRoute catalogue).
  • 27 sessions in the window: 15 cron, 3 subagent (the AI-Newsletter research fan-out), 6 desktop (voice-chat-backend research 11:28–11:32; incoming-phone-call tests 13:04 / 13:07 / 13:15), 2 acp (0 messages), 1 oneshot memory-capture probe. Fleet unchanged at 28 jobs (25 enabled / 3 disabled); 11 pin custom:omni-101 / smart-route, 8 carry no pin, 6 free-stack, 2 custom:omni, 1 deepseek-paid-smart (this audit).

Recent Changes (2026-10-22, nightly audit; live probes 2026-10-21 23:59 → 2026-10-22 00:20 AWST)

  • Hermes Agent updated to 0.22.3. hermes update ran 2026-10-21 22:47:25 → 22:57:38 AWST (action 9eed3a99e799347c691ea52a72e122a1), from pre-update tree 0.22.2 / d923a5eb487a17c0… to commit 5b147ab036d6… (detached, install stamp displayVersion 0.22.3, built 2026-10-21T14:57:38Z). Stages plan / apply (git) / deps / build all succeeded, but snapshot = skipped (disabled by updates.pre_update_backup (mode: off)). Gateway restarted via systemd — pid 868087, phase: running, api_server: connected, heartbeat 2026-10-22 00:13 AWST — and the dashboard was respawned by argv on :9119 (pid 869852). Post-run advisories, neither yet acted on: rebuild the search index (hermes sessions optimize-storage) and Preparing pinned cua-driver (Computer Use)… (no completion line in the log). → systems/hermes-agent
  • The only cron failure of 2026-10-21 is a new failure mode. Window: 171 completed / 1 failed (+ this audit). dca8482e4f76 (Daily Marketing Job Radar, 07:00 weekday, custom:omni-101 / smart-route) failed at 07:40:48 after 40 m 36 s on RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch filters reset after 19m 20s — not the 503 resource-pressure string (zero such rows on 10-21; last 2026-10-19 09:02) — while the sibling 07:00 job 85edf6772bca completed normally. Incident dca848_014affeb6954 alerted, unclosed; the run produced no output (cron_incomplete_no_output, 82 msgs / 47 tool calls). → incidents/2026-10-21-omniroute-pre-dispatch-filter-skip
  • Incidents: 1 closed / 1 opened; unclosed stays 9 (8 alerted / 1 detected). 08eb86_472c7b609c80 resolved 2026-10-21 08:37:21 after the job's 08:30 slot ran clean, which also cleared 08eb86f02d37 from the non-clean table. Non-clean last_status stays 9 (7 error / 2 delivery_failed), one swap in/out.
  • 2026-10-20 settled at 169 completed / 1 unknown — the orphaned 08eb86f02d37 run. The previous audit's "168 / 0" was taken before the 23:59 slot landed. → incidents/2026-10-20-gateway-unclean-exit-orphaned-run
  • Config backup: 76 commits now unpushed. The 2026-10-21 11:00 run created commit a8486a20 and the push failed for the 75th consecutive night; Telegram notice message_id 8449 was delivered, AgentMail failed. → incidents/2026-08-08-gitea-config-backup-push-dead
  • 12 sessions in the window, all cron — no interactive sessions. Fleet unchanged at 28 jobs (25 enabled / 3 disabled); 17 pin custom:omni-101, 7 carry no pin, 1 pins custom:omni.

Recent Changes (2026-10-21, 03:00 stocktake; live probes 03:00-03:12 AWST)

  • Scheduled wiki maintenance ran clean. wiki-maintain.sh completed all three steps — skills-index regenerated (346 → 347 skills, new australian-tax-return-review under Finance), freshness audit written (raw/stocktake/freshness-20261021.txt), Quartz rebuilt (596 files) and deployed, hermes-wiki-serve.service restarted. Job 1a8e90ea2271 is active, last_status: ok, pinned custom:omni-101 / free-stack.
  • Tailscale node inventory corrected — the table had drifted again. octop (100.92.163.122) and springroll (100.100.153.125) were missing from the table entirely; openmausbot / openmausbot-vm appeared in prose but not in the table body; omarchymac had gone offline (was online); every offline age was stale. Rebuilt from tailscale status --json keyed on DNS name, not HostName — the JSON HostName for the freshrss node is ct260, and both OmniRoute nodes report the hostname omniroute, which collides silently. Now 51 non-funnel nodes (50 IPv4 peers + hermes). → infrastructure/tailscale
  • Two Uber Eats monthly reports were missing from the published wiki and have been recovered. The generator wrote only to ~/wiki/reports/uber-eats/ (the ~/wiki repo), never to the canonical agent-estate-wiki tree — so 2026-08 and 2026-09 never reached the published site. Root cause fixed: the generator now targets the canonical tree and mirrors to the legacy one, and now emits YAML frontmatter (without it the pages landed in the audit's "no updated DATE" bucket and Quartz could not read the page date). Both months verified against ~/.hermes/uber_eats_report.json; the 2026-08 previous-month column corrected from $0.00 to $264.51 / 11 orders. → reports/uber-eats
  • Two incident pages re-verified live and annotated: incidents/2026-08-15-skills-index-drift (its named regenerate-skills-index.sh no longer exists — regeneration now lives in wiki-maintain.sh step 1/3) and incidents/2026-09-02-sftpgo-outage (CT299 running, health endpoint HTTP 401 = auth required, service up).
  • One page left stale on purpose: reports/communities-wa-social-media-audit-2026 (54d) — its source document /home/hermes/.hermes/cache/documents/doc_3cc87da4957f_deep-research-report.md no longer exists, so the baseline cannot be re-verified against a live source. Flagged, not edited.

Recent Changes (2026-10-21, nightly audit; live probes 2026-10-21 00:03-00:15 AWST)

  • A clean day: 2026-10-20 closed 168 completed / 0 failed (plus 1 unknown, plus this audit) — the first zero-failure day since 2026-10-12, ending a seven-day run of failures. The omni-101 503 fault did not recur (the last failure row carrying the resource-pressure string remains 2026-10-19), and zero new incident rows were opened across the whole window; 4 closed, so unclosed rows fall 13 → 9 (8 alerted / 1 detected). → incidents/2026-10-16-omniroute-503-resource-pressure (status back to monitoring)
  • A gateway unclean exit orphaned a cron run, and nothing reported it. 08eb86f02d37 (Gmail Action Digest, 08:30 slot) was claimed 08:31:03 and closed as unknown: “Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.” The previous gateway life (pid 2496376, started 2026-10-17 05:14) was recorded UNCLEANLY at 09:09:01 — last heartbeat 08:50:28, RSS 394 MB, 12.2 GB free, suspected_oom=False. delivery_outcome is NULL and no delivery row exists for the slot, so the ledger offers no evidence the digest arrived; no incident row was created, and the job table still shows the previous slot's 503. First recorded orphan of its kind; first unclean exit since 2026-09-21. → incidents/2026-10-20-gateway-unclean-exit-orphaned-run
  • ⚠ A green cron row is not evidence the work happened — second instance, and this one is 2.5 months old. e27da6c34bbb (Hermes Config Backup to Gitea, daily 11:00) reads last_status: ok, last_error: null, and its 2026-10-20 execution reads completed — while nothing has reached Gitea since 2026-08-07 11:00:47. The repo is 75 commits ahead of origin/main; backup.log ends could not read Username for 'https://gitea.kangaroo-eel.ts.net' → ✗ Push to Gitea FAILED → --- Backup complete: FAIL ---. origin is HTTPS with no credential in the URL and no helper, so the non-interactive push cannot authenticate: no off-box ~/.hermes backup since 2026-08-08. Telegram failure notices have been firing all along (last 2026-10-20 11:00, message_id 8447); the AgentMail step failed. → incidents/2026-08-08-gitea-config-backup-push-dead, infrastructure/backups
  • The global model default moved off OmniRoute. model.provider: custom → nous (https://inference-api.nousresearch.com/v1), model.default: smart-route → meituan/longcat-2.5-preview:free (not in place 2026-10-09 19:42; in place by 2026-10-20 11:00). Pins are unchanged — 20 of 28 jobs still pin OmniRoute (smart-route ×11, free-stack ×6, deepseek-paid-smart ×1, custom:omni ×2) — but the 8 jobs with no pin now call Nous direct instead of the router, so the fleet runs two upstream paths for the first time. → systems/current-profile, systems/scheduled-tasks
  • A user question was answered with a canned non-answer, and the request is still outstanding. Desktop session 20261020_183748_6d40ea (2026-10-20 18:37, model: minimax/minimax-m3.1-flash:free) asked how to use Nous OAuth dashboard auth for the gateway and whether a gateway token could be configured; the turn ended reason=rebuilt_restart_limit_exceeded … response_len=0 and the only assistant message is “Your request was not processed. Send it again if you still want me to carry it out.” Nothing was configured in response to that request — it needs re-asking. → incidents/2026-10-18-degenerate-stream-canned-substitute
  • dashboard.basic_auth now carries a secret and session_ttl_seconds: 43200 (present 2026-10-21 00:0x). The plugin's “no 'secret' configured; generating a random per-process signing key … sessions will not survive a restart” warning last appeared 2026-10-20 18:45:14, so dashboard sessions should now persist across restarts.
  • Interactive sessions returned after weeks of cron-only windows: talaria-ios (10:49 → 19:27, 25 messages, 12 tool calls, “Get Australian tax return done”, provider: custom:command-code) and the desktop session above. Non-clean last_status 12 → 9 (7 error / 2 delivery_failed); fleet unchanged at 28 jobs / 25 enabled / 3 disabled. → systems/scheduled-tasks

Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST)

  • The omni-101 503 resource-pressure fault returned 44 hours after the incident page was downgraded to monitoring — same node, same error string, same blast radius. 2026-10-19 closed 159 completed / 13 failed against 10-18's 168 / 1 and 10-17's 163 / 5. 9 genuine model-call failures across 8 jobs all read RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. on custom:omni-101, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs, seven jobs). The remaining 4 failed rows are the Mnemosyne Watchdog quoting the coverage monitor's error: the same derivative double-count as burst 1. → incidents/2026-10-16-omniroute-503-resource-pressure (status back to active)
  • Single pin, verified this time. Every job that failed in burst 2 reads provider: custom:omni-101, model: smart-route in jobs.json as of 2026-10-20 — there is no alias diversity left to fail over to. ⚠ The burst-1 note recording a wider mix (deepseek-paid-smart, free-IA, dark) no longer matches jobs.json; it needs confirming with Anthony before either reading is relied on.
  • The node stayed green through all of it. omniroute-1 (100.78.63.20:20128) answered /api/health → 200 {"status":"ok"} and /v1/models → 401 at 2026-10-20 00:03, before, during and after the burst. Reachability probes cannot see this failure class — only job-level error text can.
  • c0a9fa314542 (GitHub Trending Radar) lost its scheduled Monday slot (0 9 * * 1, 09:01) to the 503, so this week's roundup never ran and its revoked-token status is untested for a second consecutive Monday. Its next real test is Mon 2026-10-26 09:00.
  • Non-clean last_status moved 7 → 12 (10 error, 2 delivery_failed) as of 2026-10-20 00:09. Newly dirtied: dca8482e4f76, 85edf6772bca, 08eb86f02d37, c0a9fa314542; deepened: 0e519874d280 and ecc1fc13655a (both streak 2). fad5a86979d4 added a delivery_failed (bare-telegram target, thread 34795 not found — no incident row created for it). b63ed5817212, 96eb95ce12f2 and ea79a2107e36 all cleared on their next slots. Unclosed incident rows: 13 (12 alerted / 1 detected). → systems/scheduled-tasks
  • Unchanged otherwise: fleet still 28 jobs / 25 enabled / 3 disabled; CT255 Mattermost (100.96.100.255:8065) still unreachable; 15 sessions in the window, all cron — no interactive sessions.

Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)

  • A model stream started repeating itself — and Hermes delivered the placeholder to Telegram before it aborted. The FreshRSS Midday slot (ecc1fc13655a, 12:00) ended failed at 12:08:25 with RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response. The run had already written and delivered a canned line — "Your request was not processed. Send it again if you still want me to carry it out." — at 12:08:11 (delivery_outcome: delivered), so the digest topic received a non-answer that does not read as an error. A delivered message is not evidence of delivered content. Open question: should the guard suppress delivery or re-queue, instead of substituting text on the job's normal route? → incidents/2026-10-18-degenerate-stream-canned-substitute
  • Window tally: 10-18 = 168 completed / 1 failed — that row is the only failure of the day; 10-17 = 163 / 5, and 10-19 so far 5 / 0 plus this audit. Non-clean last_status 8 → 7; open incidents 8 → 7 (6 alerted / 1 detected), with 1 opened and 2 resolved in-window (cbfe434a3b5e 00:42:54, 1a8e90ea2271 03:01:22 — both auto-closed by their own next clean slot). 11 sessions in the window, all cron; no interactive sessions. → systems/scheduled-tasks
  • The 10-18 03:00 stocktake landed and closed the two-day gap the last audit flagged: 1a8e90ea2271 exited 0, raw/stocktake/freshness-20261018.txt exists, systems/skills-index was regenerated (346 skills / 70 categories; freshness 47 stale / 41 fresh / 7 no-date) and committed (dd53e5f).
  • Mattermost's cause is finally known: CT255 is stopped. The adapter's 5-minute reconnect loop is unchanged (attempt 471 at 00:14:07; mattermost connect timed out after 30s), the peer is offline (last seen 29d), and the host answers neither TCP :8065 nor ICMP from CT460. The 10-11 entry's "cause not determined" is now superseded. → systems/messaging-integrations
  • New page: systems/push-notifications — and the push bridge is broken, and has been for a long time. status.json says connected: false with connection refused (nothing listens there on this machine): the bridge dials HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net, which resolves on this host to 127.0.1.1 via /etc/hosts — where nothing serves 443 — while the gateway itself is healthy (0.0.0.0:9119, / → 302) and the tailnet Serve route is intact. The error string appears 23,469 times in the retained logs. Fix not applied (change the dial target, or the name resolution). → systems/push-notifications
  • One more standing alert, unchanged: the GitHub Trending Radar delivery token is still dead (getMe → 401, re-verified 10-18), and its next exercise is Mon 2026-10-19 09:00. Not repaired — a replacement bot token is Anthony's call. → systems/scheduled-tasks

Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)

  • The omniroute-1 node is up — and this time it shed load instead of refusing connections. Three days after the 34-hour TCP blackout, the same node produced a second, different failure mode: 9 model calls across 7 jobs died with RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. between 10-16 08:00:37 and 10-17 03:00:28 (~19 h), while /api/health answered 200 {"status":"ok"} and /v1/models answered 401 before, during and after (re-probed 10-18 00:33). → incidents/2026-10-16-omniroute-503-resource-pressure
  • It cleared itself and the fleet is green again. 10-16 = 165 completed / 8 failed, 10-17 = 162 / 5 (all five before 03:01; a 2026-10-20 DB re-read gives 163 / 5 — one 10-17 row was counted twice at the time), 10-18 so far 7 / 0 — no failure row of any kind since 10-17 03:00. Every job that failed re-ran clean on its next slot: b63ed5817212 10-17 06:01, 08eb86f02d37 10-17 08:31, ecc1fc13655a and ea79a2107e36 both 10-17 12:02, each of those auto-closing its own incident.
  • Fleet count fell 29 → 28: the Mosquito Tracker was removed. 1b31187397a5 is absent from jobs.json (file updated_at 2026-10-18T00:36:17+08:00); its last execution was 10-16 15:00 (completed, delivered). The three disabled jobs are unchanged — 25 enabled / 3 disabled.
  • Non-clean last_status: 12 → 8 — and seven of the eight cannot clear until their next slot, because every one of them is a weekly, Mon/Thu or monthly job: 1b23c042e17d (error, streak 2, last failure 10-16 08:03 with the 503, next Fri 10-23 08:00), 0e519874d280 (Mon 10-19 08:00), c8d52904e1d6 (Thu 10-22 18:00), 06a0a1c8704f (Thu 10-22 10:00), and the monthlies 0e753e9720fd and 066bb5191280 (both 11-01). The eighth is cbfe434a3b5e — this audit, whose 10-17 run died on the 503 and whose 10-18 run clears it. Open incidents 12 → 8 (5 alerted / 3 detected); 9 opened and 6 closed in-window. → systems/scheduled-tasks
  • The watchdog double-counts — 4 of the 13 failure rows are one sentinel. 96eb95ce12f2 exited 1 on 10-16 12:15 + 12:30 and 10-17 00:15 + 00:30, each row quoting the coverage monitor's own 503 error. Read failures per job, not per row. → systems/mnemosyne
  • Second consecutive stocktake hole. The 10-17 03:00 slot failed (503), so raw/stocktake/freshness-20261017.txt does not exist and systems/skills-index.md is stamped 10-16 (10-15 is missing too, from the storm — RuntimeError: Connection error.). The 10-18 03:00 slot closes it; nothing was regenerated by hand, because a manual run would race the scheduled one.
  • Fifth consecutive window with no interactive session — 30 sessions across the two days, all cron, and zero non-cron sessions since 2026-10-15. Every finding here comes from live system state, not conversation.
  • The GitHub Trending Radar is dead at the delivery end. The Telegram bot token in ~/workspace/github-trending-radar/config.json returns HTTP 401 on getMe (probed 10-18 00:35); the job's GitHub PAT still answers 200, so only the delivery credential is gone. The cron row stays green because last_status was set by the 10-12 09:00 run (last_run_at 2026-10-12T09:00:12). Next exercise: Mon 2026-10-19 09:00. Not repaired — a replacement bot token is Anthony's call.

Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST)

  • ⚠ A fleet-wide 34-hour cron blackout just ended — omniroute-1 was refusing TCP connections. 10-14 = 160 completed / 12 failed and 10-15 = 151 completed / 24 failed, against a clean 10-10→10-12 (175/0, 172/0, 175/0) and 10-13 (167/1). Root cause: httpcore.ConnectError: [Errno 111] Connection refused against http://100.78.63.20:20128 (provider custom:omni-101), surfacing in the cron DB as openai.APIConnectionError: Connection error. → RuntimeError: Connection error. Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. 24 genuine model-call failures on 12 jobs + 12 derivative watchdog reds = 36 failure rows across 13 jobs. → incidents/2026-10-14-omniroute-node-connection-refused-storm
  • Everything model-calling is pinned to that one node — there is no failover anywhere in the fleet. All 18 enabled agent-mode cron jobs run on custom:omni-101 (smart-route ×10, free-stack ×6, deepseek-paid-smart ×2). The only other model-capable pin, custom:omni (CT600 omniroute:20128), carries just the no_agent job e530c89f2be0, whose provider field is vestigial; d40085da631c is disabled. Completing runs inside the window: 211 — every single one script-only. Zero model calls succeeded for 34 hours. → systems/omniroute, systems/scheduled-tasks
  • The cost was silence, not errors. Digests that never went out: Tech & AI, Daily Marketing Job Radar, Political News, Gmail Action Digest ×2, FreshRSS Midday ×2, AI Newsletter Curation for Patrick; the weekly Perth New Restaurant was missed outright (next chance 10-22). The 10-14 23:59 Wiki Audit itself died on the same fault, so the 10-14 window was never audited — this entry spans two days. → systems/scheduled-tasks
  • Recovered, verified live. GET /v1/models on 100.78.63.20:20128 → HTTP 401 in 10 ms (up, auth required); same on omniroute:20128 / 100.88.81.19. First green model-call job: 1b31187397a5 at 10-15 22:01:56; then b63ed5817212 (10-16 00:02:29) and ea79a2107e36 (10-16 00:09:48).
  • The backlog unwinds on schedule rather than by hand. Non-clean last_status: 3 → 12 (9 new storm rows; three cleared as their slots came round), open incidents 3 → 12 (9 alerted / 3 detected). The two longest-lived — 1b23c042e17d (Fri 08:00 slot) and 0e753e9720fd (1st 09:00) — pre-date the storm. → systems/scheduled-tasks
  • Both Mnemosyne monitors sit inside the blast radius. Health (12 h) and Embedding Coverage (6 h) both failed on every slot, so no coverage figures were refreshed for ~34 h — the last explicit read remains 2026-10-12 (working 96.7 %, episodic 99.4 %). The script-only Mnemosyne jobs were unaffected. → systems/mnemosyne
  • Self-healing gap, not a manual repair. The failed 10-15 03:00 stocktake left raw/stocktake/freshness-20261015.txt unwritten and systems/skills-index.md stamped 10-14; the 10-16 03:00 slot closes it, so nothing was regenerated by hand here.
  • Fourth consecutive window with no interactive session — 36 sessions across the two days, all cron.

Recent Changes (2026-10-14, 03:00 stocktake; live probes 03:01–03:1x AWST)

  • ⚠ A second, unclustered Proxmox hypervisor exists — m93p at 100.104.51.98, PVE 9.2.0. DISCOVERED this pass. pvecm nodes is empty on both hosts (no cluster), and guest IDs collide: m93p CT221 is gitea, pve CT103 is cliproxyapi, and both worlds share CT100/CT221/CT245 numbers. It had been in the Tailscale node list as an unremarkable online peer with no wiki page recording it as a hypervisor; a tailnet-wide :8006 scan returned only m93p and pve, which is how it surfaced. Live guests on m93p include gitea (CT221), ntfy (CT1241), freshrss (CT1260), openweb (CT245), vaultwarden, obsidian-livesync, rustdesk, homebridge, litellm (CT500), omniroute (CT101), paseo (CT1261), mailflow, social-dashboard, docker-01, rclone, freellmapi; nightbridge stopped. → infrastructure/proxmox, infrastructure/network-map
  • Four guests the wiki placed on pve are actually on m93p. CT221 (gitea), CT241 → CT1241 (ntfy), CT260 → CT1260 (freshrss) and CT245 (openweb) are absent from live pct list on pve, yet all four services answer. /etc/pve/lxc/ on pve holds only 260.conf.bak and no vzdump log references them — so they were never deleted, they were never on this host. Every prior audit since 2026-10-11 carried the error forward. Always qualify a guest with its host. → infrastructure/proxmox
  • VM 282 ubuntu-server-xfce is running, not stopped as recorded. → infrastructure/proxmox
  • Tailscale node inventory rebuilt: 45 non-funnel nodes (plus 23 funnel-ingress-node IPv6 peers, excluded). The 2026-10-11 table had drifted badly — it listed 30 nodes and was missing 15, including m93p, pve, omniroute-1, paseo, qwenpaw, rustdesk, vaultwarden, social-dashboard, openweb, omarchybook, sftpgo, rclone, omni, runtipi. Several states were wrong too (anthonys-mac-mini is online on a different IP; omarchy-macmini/omarchy-quattro are not live peers; anthonys-iphone is online not active). Serve routes on CT460 re-verified exactly as recorded — 13 routes, :8443/webhook Funnel, :9443 tailnet-only. → infrastructure/tailscale
  • Mattermost CT255 confirmed stopped, consistent with its Tailscale peer offline 24d — unchanged finding, re-verified. → infrastructure/proxmox
  • Camofox recovered from its 2026-10-12 memory ceiling: camofox-browser.service now reports Memory: 29.9M (high: 2G, max: 3G, swap max: 1G, available: 1.9G, peak: 485.8M) with 16 tasks, active since 2026-10-13 23:42:55. The containment limits held; the workload was the fault. → systems/browser-backend
  • Stocktake artifacts regenerated: systems/skills-index.md = 346 skills / 70 categories (the known 347-on-disk off-by-one persists); raw/stocktake/freshness-20261014.txt = 48 stale / 38 fresh / 7 no-date of 93 pages, identical in shape to 10-13.
  • Quartz rebuild still inert (no quartz.config.ts, no npx) — public/ was not refreshed. Carried open item.

Recent Changes (2026-10-13 → 2026-10-14, nightly audit; live probes 2026-10-14 00:00-00:30 AWST)

  • The four-day clean streak broke: 2026-10-13 = 167 completed / 1 failed (10-12 finalised 175 / 0, 10-11 172 / 0, 10-10 175 / 0). The failure is 1a8e90ea2271 (wiki-stocktake-maintenance, 03:00) on RuntimeError: HTTP 404: No active credentials for provider: opencode — the third failure of that job in six days, all three on the opencode path (10-08 403 free-tier, 10-09 timeout, 10-13 404). New incident 1a8e90_d11698bd7275 (detected, streak 1, not alerted); it clears only on a successful 2026-10-14 03:00 run. → systems/scheduled-tasks
  • The failing run stranded its own generated artifacts. Step 1 (skills-index) and step 2 (freshness report) wrote to disk, then the agent turn died → no commit, no push. systems/skills-index.md (date-only change; 346 skills / 70 categories unchanged) and raw/stocktake/freshness-20261013.txt (48 stale / 38 fresh / 7 no-date of 93 pages, down from 54 stale) were committed by the 2026-10-14 audit. → systems/skills-index
  • executions.db retention floor moved to 2026-10-08 07:11 (MAX_TERMINAL_EXECUTIONS = 1000; 1,001 rows = 1,000 terminal + 1 in flight). Days older than the floor now undercount — 10-08 reads 126 completed / 1 failed, a partial tally.
  • Non-clean last_status rows: 3 → 4 (cleared fad5a86979d4, dca8482e4f76; added 1a8e90ea2271). Open incidents: 2 → 3. Fleet unchanged at 29 jobs / 26 enabled.
  • No interactive sessions for the third window running: 15 sessions in the window, all cron, zero interactive.

Recent Changes (2026-10-12 → 2026-10-13, nightly audit; live probes 2026-10-13 00:05-00:45 AWST)

  • A clean cron day hid a dead credential. 2026-10-12 ran 174 completed / 0 failed (third consecutive clean day; 10-11 finalised at 172 / 0) — yet c0a9fa314542 (GitHub Trending Radar) reported completed with delivery_outcome: delivered while its own Telegram send was dying on an invalid token. The delivered artifact was the agent's recovered digest, not the script's. → systems/scheduled-tasks
  • GitHub Trending Radar's bot token is REVOKED — live-verified, and the 2026-10-06 "refuted" entry was WRONG. getMe against telegram.bot_token in /home/hermes/workspace/github-trending-radar/config.json returns HTTP 401; the token is a valid 46-char shape, so revoked rather than malformed, and no valid replacement exists on the box. The fetch path is healthy (100 repos, 26 cache files in 24 h), so only digest.py's send() fails. Correction: the 2026-10-06 entry below treated two green runs as proof the token worked; a green cron row only means the agent finished, and neither run exercised the failing call. Fix = new @BotFather token → config.json → re-run getMe. Credential present, value not recorded. → systems/scheduled-tasks
  • Open cron incidents 5 → 2. Closed on 2026-10-12: fad5a8_a13bbc837e1c + fad5a8_c80b59a907f0 at 03:05:39 (both deepseek-v4-flash, cleared by fad5a86979d4's first clean scheduled run — 03:00:45 → 03:05:47, 448 unique birthdays, delivered) and dca848_23fcd2b18aaa at 07:33:41 (cleared by dca8482e4f76's own 07:00 run, delivered to topic 1030 after a 32-minute run). Survivors: 0e753e_e7f8194a7d06 (detected since 2026-10-01 — cannot clear until 2026-11-01, the job has no retained executions at all) and 1b23c0_c35703f637b8 (alerted since 2026-10-09, next slot Fri 2026-10-16 08:00). Table: alerted 1 / detected 1 / closed 8 / resolved 66. Fleet 29 jobs — 26 enabled / 3 disabled; non-clean last_status 5 → 3 (0e753e9720fd, 1b23c042e17d, 066bb5191280). → systems/scheduled-tasks
  • Camofox hit its memory containment ceiling inside a cron run (NEW). The 2026-10-12 07:01 Daily Marketing Job Radar drove camofox-browser.service to 2.5G with swap fully consumed, available: 0B, 383 tasks (drop-in MemoryHigh=2G/MemoryMax=3G/MemorySwapMax=1G); /tabs and page evals hung. The agent restarted the service at 07:23:50 and finished one tab at a time, and the job still delivered. A green cron row does not mean the browser layer was healthy — check the unit's peak/swap/tasks when a browser-heavy job runs long. → systems/browser-backend
  • Browser page corrected against a live probe. REST API is :9377, not :9093 (nothing listened on 9093); :9378 does not exist; Camofox is a systemd --user unit on the Hermes host (CT460), not CT450. Containment drop-in documented. → systems/browser-backend
  • Mnemosyne healthy at 12:02 — working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 % (both above the 95 % floor, no reindex needed); orphan embeddings 0; FTS5 aligned exactly (786/786, 1056/1056, 876/876); 0 unconsolidated; 1 unresolved conflict; disk 15 GB free (81 % used). → systems/mnemosyne
  • Window shape: 18 sessions, all cron, zero interactive — the other bookend to the previous window's talaria-ios research session. The 03:00 stocktake 1a8e90ea2271 ran clean and pushed 0eee5f1 (freshness 54 stale / 32 fresh / 7 no-date of 93 pages; skills index 346 / 70 categories — the known 347-vs-346 off-by-one persists). The execution log's retention floor has moved to 2026-10-07 08:15, so 10-07 rows are now pruned.

Recent Changes (2026-10-12, 03:00 stocktake; live probes 2026-10-12 03:0x-03:1x AWST)

  • 02 stocktake corrections from live probes. systems/nanobot was DOWN as of 2026-08-07; its web UI on CT333 is back up (HTTP 200 on 100.122.189.81:8766), though the OmniRoute port 8900 endpoint is still closed so status stays degraded. systems/home-assistant confirmed up: HTTP 200 on 100.68.139.5:8123, VMID 100 running, Tailscale node offering an exit node — confidence raised low → medium. → systems/nanobot, systems/home-assistant
  • Freshness snapshot: 54 stale / 32 fresh / 7 no-date of 93 pages (threshold >21 days). The bulk of the stale list is the 2026-07-22 seeding cohort (concepts, runbooks, comparisons) and is structural, not a regression. → raw/stocktake/freshness-20261012.txt
  • Skills index regenerated: 346 skills / 70 categories, unchanged count from the previous run. → systems/skills-index
  • Tailscale routes re-verified against tailscale serve status — the CT460 table in infrastructure/tailscale matches live output exactly (13 tailnet routes + :8443/webhook Funnel + :9443 Vite dev). Two nodes joined since 2026-10-11: omarchy (100.73.227.103) and omniroute-1 (100.78.63.20, both active).

Recent Changes (2026-10-11 → 2026-10-12, nightly audit; live probes 2026-10-12 00:0x-00:2x AWST)

  • The cron fleet is clean for a second consecutive day: 2026-10-11 ran 171 completed / 0 failed (+1 running = this audit), after 10-07 1 / 10-08 2 / 10-09 15 failed. 10-10 finalises at 175 / 0. Fleet 29 jobs — 26 enabled / 3 disabled; non-clean last_status steady at 5; 0 incidents opened, 0 closed — 5 unresolved (3 alerted, 2 detected). The two streak-1 jobs (dca8482e4f76, 1b23c042e17d) carry Friday 2026-10-09 timeouts and have not had a scheduled slot since. → systems/scheduled-tasks
  • The 2026-10-11 00:08 "memory capture stalled" EXCEPTION was a timezone mislabel, not a capture fault. working_memory.created_at is naive UTC; the monitor compared it against AWST now and reported a 2.7 h gap as 10.7 h. Capture verified healthy: 786 rows, +2 in the window, coverage 98.3 % working / 99.6 % episodic. → systems/mnemosyne
  • Proxmox storage documented for the first time: /mnt/ssd0 is the pressure point — 808 GiB used, ~63 GiB free (93 % by df, 88.1 % by pvesm); SSD1 86.1 %, fred-qnap 76.8 %, iMacHDD inactive. → infrastructure/proxmox
  • First retained research answer filed: fastest VNC clients (asked 2026-10-11) — the server-side encoder is the real bottleneck, TigerVNC native viewer for standard RFB, -CompressLevel is a client parameter, RealVNC's "4.7×" is self-referential. → queries/retained-research-answers
  • Only one interactive session in the window (talaria-ios, 19:13 → 19:51, 41 messages); the other 17 were cron. The 03:00 stocktake 1a8e90ea2271 completed on schedule (03:00:40 → 03:11:29) and committed d981e30.
  • Stale labels corrected: incidents/2026-08-18-mnemosyne-capture-pipeline moved from active to superseded (evidence above); the index's count of open 2026-10-09 storm incidents trimmed to the 2 that are genuinely open.

Recent Changes (2026-10-11, 03:00 stocktake; live probes 03:00-03:30 AWST)

  • OmniRoute CT600/CT601 roles are INVERTED. CT600 omniroute (100.88.81.19) is running and serving :20128 (combo API returns live combos); CT601 omni (100.93.204.4) is stopped with onboot=0 (Tailscale peer offline 54d). Hermes resolves custom:omni → omniroute:20128 → CT600. → systems/omniroute
  • A third OmniRoute node exists: omniroute-1 at 100.78.63.20:20128, live (401 = auth-required), registered in Hermes config as provider omni-101 (5384 models, default smart-route). It carries skills_hub (model free-stack) — the same provider family implicated in the 2026-10-09 timeout storm. Not present in pct list; identity unconfirmed. → systems/omniroute
  • Two Ignis instances exist, on different hosts. CT460 ignis (port 8080, vault /home/hermes/wiki/agent-estate-wiki, serving 200) and CT200 ignis-ignis-1 (image ignis-server:latest, host port 8092, vault /opt/ignis/vaults, serving 200). Both healthy; they are separate deployments. → systems/ignis
  • Tailscale route drift fixed: /agent-estate targets bare http://127.0.0.1:8769 (page had .../agent-estate-role-dashboard.html); a tailnet-only :9443 → 127.0.0.1:5173 Serve was undocumented. CT201's live routes are 8085-based plus /steel (5173) and /steel-api (3000) — the old :8443 → localhost:8082 Homepage row is wrong. (The /steel and /steel-api routes were deleted on 2026-10-29 — see systems/browser-backend.) → infrastructure/tailscale
  • Proxmox inventory rebuilt from pct list + qm list: CT475 agentzero and VM110 omarchy were missing entirely; 282 ubuntu-server-xfce corrected to stopped; dead rows removed (310, 500, 104, 106, 101). VM403 open-claw and VM401 hermes-old are absent from qm list — flagged unconfirmed, not deleted. → infrastructure/proxmox
  • Nanobot verified on CT333: nanobot-gateway (PID 2259048) alive on 8765/8766. It is not a systemd nanobot.service/gateway.service (both probes return inactive — wrong unit names); it runs as the nanobot user service plus the gateway process. Port 8900 is no longer listening — previous "Serve API" claim flagged unverified. → agents/nanobot
  • Docker services page corrected: the old "Docker runtime not available" line was a CT460 socket artifact (/var/run/docker.sock is root:docker srw-rw----; hermes user not in group docker; no passwordless sudo in cron). Real fleet: CT200 holds ~50 containers, CT333 holds 9. → infrastructure/docker-services
  • Mattermost CT255 confirmed stopped, consistent with its Tailscale peer being offline 21d. Root cause of the unreachable endpoint still undetermined.
  • ⚠ mycompose-db-1 on CT200 is in Restarting (1) — crash-looping. Flagged; root cause NOT investigated this pass.
  • Stocktake artifacts: skills-index regenerated (346 skills / 70 categories), raw/stocktake/freshness-20261011.txt = 62 stale / 24 fresh / 7 no-date of 93 pages.
  • Quartz rebuild still inert (no quartz.config.ts) — public/ was not refreshed. Carried open item; unchanged from prior days.
  • 62 pages remain >21d stale and were not corrected this pass: they were not probed against a live source, so they stay flagged rather than rewritten.

Recent Changes (2026-10-10 -> 10-11, nightly audit of the 2026-10-09 -> 2026-10-10 window; probes 2026-10-11 00:14-00:20 AWST)

The manual 2026-10-10 stocktake entry follows beneath this one.

  • The cron fleet is clean again: 174 completed, 0 failed on 2026-10-10 (AWST day, executions.db) - after 10-07 1 / 10-08 2 / 10-09 15 failures. 7 of the 10 storm incidents auto-closed during the day; 5 remain open over 3 jobs (1b23c042e17d + dca8482e4f76 timeout/alerted since 2026-10-09; 0e753e9720fd + fad5a86979d4 agent). Non-clean last_status down from 10 to 5. Job count 30 -> 29 (26 enabled / 3 disabled). -> systems/scheduled-tasks
  • The DHL monitor 62619aa6bcb3 is retired - open item #1 closed. Anthony instructed removal again on the evening of 2026-10-10; the job is absent from live jobs.json (verified 2026-10-11 00:19 by direct ID lookup). The 2026-10-07 and 2026-10-09 "deleted" reports were false - they inferred removal from missing output instead of looking up the ID.
  • The 2026-09-28 runtime-dependency class recurred - in the gateway env this time. The gateway's activate_dependencies() loads one committed uv env (795020dbf24343cd9c52df24723dca83); it was missing mnemosyne / mnemosyne_hermes plus native deps, and elevenlabs was missing from the live managed env. Repaired 21:26-21:34, verified live from 22:01:19 (Memory provider 'mnemosyne' registered (40 tools) -> activated; repeated 23:59 / 00:00 / 00:02). The 15-minute guard does not cover this env - open. -> incidents/2026-09-28-hermes-managed-runtime-deps
  • New page: systems/paseo - Paseo coding-agent daemon, user unit paseo-daemon.service, v0.11.2, 0.0.0.0:6767, running since 2026-10-10 14:47:28. Pitfalls recorded: daemon run vs daemon start, /usr/bin/paseo is 0.7.2 while the daemon is 0.11.2, and a truncated cline binary presenting as a provider fault.
  • Mattermost unreachable from CT460 - configured endpoint 100.98.116.14:8065 gives 100 % ICMP loss and TCP refuse; the gateway retries every 5 minutes. Cause not determined. -> systems/messaging-integrations
  • Health-monitor follow-ups (2026-10-10): "capture silent" was a false alarm (cron-only window); check_embedding_coverage.py now carries mode 0755; 13 orphan memory_embeddings rows (oldest 2026-08-22) remain unverified. -> systems/mnemosyne

Recent Changes (2026-10-10)

  • The 2026-10-09 timeout storm is over — the fleet is green again. Window tally (executions.db, 2026-10-09 00:00 → 2026-10-10 00:00 AWST): 163 completed, 15 failed. Every failure was RuntimeError: Request timed out. on the routing aliases behind custom:omni-101, confined to 00:00 → 15:00; all runs from 18:00 onward completed. 10 open incidents remain (ecc1fc_58b6c38296f3, 08eb86_634f1a296f78, 1b23c0_c35703f637b8, 85edf6_18bedbeec5cd, dca848_23fcd2b18aaa, 1a8e90_dc6b2d40ec44, 1a8e90_789224ad589c, 0e753e_e7f8194a7d06, fad5a8_c80b59a907f0, fad5a8_a13bbc837e1c). → incidents/2026-10-09-cron-provider-timeout-storm
  • 1a8e90ea2271 (stocktake) is running right now, not failed. Its 2026-10-09 03:00 run timed out before the generator produced output (so no freshness-20261009.txt exists — the two-day artifact gap stands); the 2026-10-10 03:00:17 run is running at audit time. failure_streak: 2 until it completes. This stocktake (run manually by this audit) regenerated systems/skills-index.md (346 → 346 skills / 70 categories, frontmatter dates refreshed) and wrote raw/stocktake/freshness-20261010.txt — 62 stale, 23 fresh, 7 no-date of 92 pages. See systems/scheduled-tasks.
  • Skills count reconciled: 347 SKILL.md on disk vs 346 reported — unchanged off-by-one (the generator de-dups the nested productivity/obsidian-git-sync/obsidian-git-sync/ duplicate). Carried open item.
  • 8 jobs carry a stale last_status: error — 0e753e9720fd (streak 0), fad5a86979d4 (streak 0), dca8482e4f76 (streak 1), 1b23c042e17d (streak 1), 85edf6772bca (streak 1), 08eb86f02d37 (streak 1), ecc1fc13655a (streak 1), 1a8e90ea2271 (streak 2). Two are pre-repin stale fields (streak 0), not live failures. Job count unchanged: 30 (27 enabled / 3 disabled).
  • DHL monitor 62619aa6bcb3 still enabled — re-probed 2026-10-10: enabled: True, paused_at: null, schedule 0 8,11,14,17,20 * * *. Anthony instructed its deletion on 2026-10-09 21:24 and a session reported it deleted. The parcel (8052838382) was Delivered 2026-10-07, so the monitor is redundant. This audit did not mutate cron state — retirement is open item #1 in systems/scheduled-tasks.
  • Quartz rebuild step confirmed still inert (no quartz.config.ts, npx present but unreachable for quartz) — the public/ static site was not refreshed by this stocktake. Skills-index and freshness artifacts regenerated and committed correctly; only the published-HTML step is a no-op. Carried open item from 2026-10-07.
  • Memory recall ranking fix still not persisted — the 2026-10-09 in-session "config saved" claim is not in live config; carried open item. Capture/vector health unchanged (760/760 working, 1,043/1,047 episodic embedded).
  • infrastructure/tailscale.md re-verified 2026-10-10 — tailscale serve status matches the page exactly: 13 tailnet Serve routes (/, /ui, /wiki, /ignis, /webui, /gitea, /jobs, /desktop, /vnc-camofox, /synergy-prep, /agent-estate, /evening-grid.html, /ignis/:8443 funnel /webhook) + the :8443/webhook public Funnel. No drift; page remains fresh (19d).

Recent Changes (2026-10-09)

  • Cron fleet: the worst day since the repin. 15 failed executions across 10 jobs on 2026-10-09 (10-07: 1, 10-08: 2) — all RuntimeError: Request timed out. on the aliases behind custom:omni-101, 00:00 → 15:00 AWST, clean from 18:00. Four incidents auto-closed, eight still open; 1a8e90ea2271 sits at failure_streak: 2. → incidents/2026-10-09-cron-provider-timeout-storm
  • The 2026-10-08 audit window was lost to the same timeout (this job's own 23:59 run); the last committed audit covered 2026-10-07. The 2026-10-09 run covers both days. → log
  • 62619aa6bcb3 (DHL Tracking — Parcel 8052838382) is still enabled even though Anthony instructed its deletion on 2026-10-09 21:24 and the session reported it deleted; it runs at 08:00 daily. Retirement is open item #1 in systems/scheduled-tasks.
  • Memory recall relevance incident — ✅ RESOLVED 2026-11-01. A stale conversational episode had outranked current-state facts on an open question. The ranking fix was persisted — the weights live in ~/.hermes/mnemosyne/config.yaml (vec_weight: 0.5, fts_weight: 0.3, importance_weight: 0.3), not in the ~/.hermes/config.yaml block the original probe read. Live tier counts: working 941 (embeddings 927/941, 98.5 %), episodic 1,124 (embeddings 1,124/1,124, 100 %), canonical 90, conflicts 175 with 11 open (extraction noise). The original regression query no longer surfaces the stale episode. → incidents/2026-10-09-memory-recall-relevance
  • Mnemosyne live counts (2026-10-09 22:27): 773 working / 1,047 episodic / 97 canonical slots / 871 facts / 152 conflicts (1 open) / 760 working embeddings / 1,043 episodic embeddings. → systems/mnemosyne

Recent Changes (2026-10-08)

  • Cron fleet green for a third consecutive day — 0 failing. Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 failure_streak counters still read 0. Window tally (executions.db, 2026-10-07 00:00 → 2026-10-08 00:00 AWST): 176 completed, 1 failed, 1 running (the nightly audit itself). 13 sessions started in the window, all source = cron — no interactive (non-cron) sessions; 12 deliveries, 0 delivery failures.
  • First detect → notify → auto-resolve incident cycle since the managed-runtime event (DHL, 2026-10-07 08:00 → 11:01). 62619aa6bcb3 (DHL Tracking, 5×/day) failed its 08:00 run inside ensure_browser() with TimeoutError: timed out — Camofox briefly unresponsive on a browser HTTP call (dhl_tracking_monitor.py:233). The failure notice went out at 08:00:58 and incident 62619a_f485ccfd4a0d (failure_type: timeout) auto-resolved at 11:01:13 when the next 3-hourly run completed; the 14:00, 17:00 and 20:00 runs also succeeded. No action needed — and the incident machinery behaved exactly as designed, which it had not been tested on since 2026-09-30. See systems/scheduled-tasks.
  • The tracked parcel arrived — the DHL monitor is now redundant. The 14:00 run on 2026-10-07 reported waybill 8052838382 as Delivered. 62619aa6bcb3 still runs 5×/day for no remaining benefit; flagged as a removal candidate rather than removed.
  • The stocktake job proved its repin for a second day. 1a8e90ea2271 completed automatically at 2026-10-07 03:00 — 346 skills / 70 categories (unchanged), raw/stocktake/freshness-20261007.txt (62 stale, 21 fresh, 7 no-date of 90 pages), committed to Gitea as 8c63ef2. Two script-level findings came out of the run: (1) the Quartz rebuild step is a no-op — no quartz.config.ts is present and npx is unavailable, so the published site is never refreshed; (2) skills-index under-counts by one — 347 SKILL.md files exist on disk while the page reports 346, because the generator de-dups the nested productivity/obsidian-git-sync/obsidian-git-sync/ duplicate away. Both are now recorded as open issues in systems/scheduled-tasks.
  • Nothing else changed in the window. The three stale pre-repin last_status: error jobs (1b23c042e17d, fad5a86979d4, 0e753e9720fd) remain — due Fri 08:00 / Mon 03:00 / 1st 09:00. The custom:omni stale-key references and the three bare-telegram targets are unchanged. No new incident page was required, and no page was removed or archived.

Recent Changes (2026-10-07)

  • Cron fleet green for a second consecutive day — 0 failing. Job count unchanged at 30 (27 enabled / 3 disabled) and all 30 failure_streak counters read 0. Window tally (executions.db, 2026-10-06 00:00 → 2026-10-07 00:00 AWST): 176 completed, 0 failed, 1 running (the nightly audit itself) — 177 rows. 14 sessions started in the window, all source = cron; no interactive (non-cron) sessions. No new incident was opened. The repin wrap-up closures at 00:10–00:12 on 2026-10-06 fall inside this window but were already recorded by the 2026-10-06 audit.
  • The stocktake job proved its repin — first fully automatic run since 2026-09-25. 1a8e90ea2271 (wiki-stocktake-maintenance, 03:00 daily) ran on schedule 2026-10-06 03:00:31 → 03:03:32 on its new custom:omni-101 / free-stack pin and completed cleanly. Its four HTTP-410 incidents (6df957a05ec6, d284168d3663, cda67de5b3e3, e31584909bd2) had already been closed at 00:10–00:11 on 2026-10-06 during the repin wrap-up. It regenerated systems/skills-index.md (344 → 346 skills), wrote raw/stocktake/freshness-20261006.txt and committed to Gitea as 33f4e95. Its stale pre-repin last_status: error is now cleared — live last_status: ok, streak 0. The daily manual wiki-maintain.sh run is no longer needed. See systems/scheduled-tasks.
  • Three jobs still carry a stale pre-repin last_status: error — 1b23c042e17d, fad5a86979d4, 0e753e9720fd (down from four). Each clears on its next scheduled slot (Fri 08:00 / Mon 03:00 / 1st 09:00); none is a live failure. The bare-telegram target warning still stands on 066bb5191280 and 0e753e9720fd (unchanged).
  • Wiki vault mirror refreshed. systems/skills-index.md in /home/hermes/wiki/agent-estate-wiki/ had drifted to 2026-10-03 / 344 skills while the canonical copy regenerated to 2026-10-06 / 346; this audit re-mirrored it along with the pages below. The two-wiki-tree drift remains a standing open item.

Recent Changes (2026-10-06)

  • Dead model pins cleared across the whole fleet — failing set 7 → 0. All seven jobs that had been failing since 2026-10-01 were repinned on 2026-10-05 23:16 → 2026-10-06 00:05. Each had been pinned to a raw model name that died upstream: cliprox/terra (HTTP 401 connection auth, 3 jobs), ollama-cloud/qwen3.5:397b (HTTP 410, retired 2026-09-25), nvidia/deepseek-ai/deepseek-v4-flash (HTTP 400, dropped from the live catalog, 2 jobs) and command-code/gpt-5.6-terra (HTTP 403 MODEL_NOT_IN_PLAN). They were moved to custom:omni-101 on routing aliases — free-stack ×6, smart-route ×1 — because the fault was raw-pin brittleness, not any one model. Every failure_streak in jobs.json now reads 0. Three were verified by live run: 79f9bb41ecf5 (23:39), 06a0a1c8704f (23:45) and dca8482e4f76 (23:56, delivered to topic 1030 — its first success since 2026-09-25). The other four are repinned but not yet re-run, so they carry a stale last_status: error until their next scheduled slot. See incidents/2026-10-05-dead-model-pin-repin.
  • The cliprox/terra connection incident is closed for cron purposes. No enabled job references that pin any more — the jobs were moved off it rather than the proxy connection being re-authenticated. See incidents/2026-10-01-cliproxy-connection-auth-expired.
  • GitHub Trending Radar's revoked-token diagnosis is refuted. SUPERSEDED 2026-10-13 — this conclusion was WRONG. c0a9fa314542 ran on schedule at 2026-10-05 09:00 and completed cleanly — a second consecutive success after the off-schedule 2026-09-30 19:22 run. No getMe → 401, and no BotFather token replacement is needed. The previous audit correctly refused to treat the claim as standing fact; it can now be retired.
  • A stale failure notice, not a live outage, opened the review. Anthony acted on a 85edf6772bca notice that had been sitting in topic 346 since the 2026-09-29 07:00 run. That job succeeded on every run from 2026-09-30 17:53 onward, and its three incidents were resolved on 2026-09-30 17:55:13; the notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. Check executions.started_at and the cron_incidents state before acting on a notice.
  • Window tally (executions.db, 2026-10-05 00:00 → 2026-10-06 00:15 AWST): 185 completed, 3 failed (all pre-repin, 03:00–07:00), 2 unknown (aborted first attempts of 79f9bb41ecf5 / dca8482e4f76, superseded by their successful re-runs) and 1 running (the nightly audit itself). No interactive sessions other than the repin session.

Recent Changes (2026-10-05)

  • Cron fleet re-verified 2026-10-05 00:01 AWST — failing set 6 → 7. Job count unchanged at 30 (27 enabled / 3 disabled). The one new failure event in the audit window is 79f9bb41ecf5 (Weekly Review, Sun 18:00), which ran on schedule 2026-10-04 18:00 and failed with the identical cliprox/terra error the incident predicted: HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. That is the last untested job on the pin, so the provider-connection auth expiry is now 3 of 3 confirmed — settled as a credential fault, not per-job. No review was produced; delivery_outcome: delivered again because the failure notice is the deliverable. See incidents/2026-10-01-cliproxy-connection-auth-expired.
  • 1a8e90ea2271 failed again — 2026-10-04 03:00 (HTTP 410). The wiki-stocktake-maintenance job remains pinned to the Ollama-retired qwen3.5:397b; no change, still needs a repin. Its script is healthy. See systems/scheduled-tasks.
  • No interactive (non-cron) sessions in the window — all 10 sessions active in the past 24h were cron. The only durable changes are cron-fleet status, reflected above.
  • Streak-number caveat for future audits: the N failures in a row figure in hermes cron list is the failure_streak counter from jobs.json, not a DB-derived count. ~/.hermes/cron/executions.db now retains only rows from 2026-09-30 onward (1,001 rows), so DB recounts undercount older streaks. Live counters 2026-10-05: 1a8e90ea2271 ×11, dca8482e4f76 ×7, fad5a86979d4 ×3. Quote the live counter, and note the retention floor.

Recent Changes (2026-10-04)

  • Both Hermes instances updated on 2026-10-03, from opposite directions. CT460 (production) was rebuilt to v0.21.5+6169.gab86a20 (upstream ab86a20a, tree mtime 10:11 AWST) and CT450 (chaos) came up from v0.19.0 to v0.21.5+6170.ga510d6c (upstream a510d6c6, tree mtime 10:22 AWST). Same semver token, different builds, separate state. CT450 now reports Update available: 3 commits behind. CT450 has no SSH — it is reached via ssh root@100.96.100.82 + pct exec 450, and hermes is not on the default pct exec PATH. See incidents/2026-10-03-ct450-hermes-update.
  • Vector recall had been silently running keyword-only — found and fixed 2026-10-03. fastembed was missing from every uv-managed runtime, so the dense leg of mnemosyne_recall contributed no candidates and recall fell back to FTS5 without erroring. Fixed the same day: fastembed 0.8.1 installed into the bootstrap-selected runtime, FASTEMBED_CACHE_PATH pinned, and hermes-runtime-deps-guard.sh extended to assert fastembed + cache path + model dir every 15 min. Live coverage 2026-10-04: working 726/752 (96.5%), episodic 1033/1039 (99.4%), 13 embeddings generated in the previous 24h. The previous "coverage gap" entry (159/1,476) is retired — it was resolved by the 2026-08-02 reindex. See incidents/2026-10-03-mnemosyne-vector-recall-fts-only and systems/mnemosyne.
  • Managed runtimes pruned 10 → 3. Only 3 uv-managed venvs remain under ~/.hermes/installs/019d0d114d22b7e6/environments/ (two created 2026-10-03 09:59/10:00, one surviving from 2026-09-28). Any note that says "ten managed venvs" is stale.
  • Cron fleet re-verified 2026-10-04 00:12 AWST: 30 jobs (27 enabled / 3 disabled), failing set unchanged at 6, no new failure inside the window. 79f9bb41ecf5 — the last untested cliprox/terra job — is due 2026-10-04 18:00 AWST. See systems/scheduled-tasks.
  • Mattermost is unreachable from CT460 right now (mattermost connect timed out after 30s, next retry in 300s, every 5 min, live 2026-10-04). Host-side condition; Mattermost delivery should be treated as down until probed.
  • Wiki maintenance gotcha worth remembering: mnemosyne doctor runs under the legacy Python 3.11 venv/, where sqlite_vec is not importable, so it prints vec0: present_but_unloadable and marks vector tiers unavailable even when the gateway is embedding fine. Judge vector health with check_embedding_coverage.py, never with doctor's vec0 line.

Recent Changes (2026-10-03)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set stays at 6, but the composition changed: 1b23c042e17d (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the identical HTTP 401 the cliprox incident predicted — HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard. That makes 2 of the 3 cliprox/terra jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. 79f9bb41ecf5 (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due 2026-10-04 18:00. See incidents/2026-10-01-cliproxy-connection-auth-expired.
  • Streaks recounted from executions.db — one number corrected: dca8482e4f76 6 → 7 (one more daily run), but 1a8e90ea2271 8 → 7 — the ×8 carried since 2026-10-01 was an overcount; that job has exactly 7 recorded runs and all 7 failed. Recount before quoting a streak.
  • A delivered notice is not a successful job. Both cliprox/terra failures are recorded delivery_outcome: delivered because the failure notice is itself a deliverable — the execution status is failed and no digest/plan was produced. Judge cron health on the status column, not the delivery outcome.
  • Resolved — 1b23c042e17d stale delivery_failed flag cleared: the 2026-09-30 retarget to telegram:-1004321904721:102 worked exactly as predicted; its 2026-10-02 run delivered (then failed on the model pin). The prediction held.
  • systems/skills-index regenerated by hand: 341 → 344 skills. The daily stocktake job (1a8e90ea2271) is still blocked on a retired model (HTTP 410, qwen3.5:397b), but its script is healthy — wiki-maintain.sh was run manually on 2026-10-03 and completed cleanly, refreshing the index and writing raw/stocktake/freshness-20261003.txt. One new skill appeared: fleet-wide-failure-triage (devops). Only the scheduled wrapper is broken; every non-model step of the stocktake now needs a manual run each day until it is repinned.
  • Work mailbox blocker documented — and it is older than it looks. A new incident page records that the work Outlook account has been unreachable since 2026-08-26, not this week: the account uses password auth while the tenant advertises only XOAuth2, so himalaya -a work dies with login not supported. The TCP/TLS path is fine — it is a tenant policy wall, not a bad password, and re-running the credential helper can never fix it. It needs an OAuth2 path. No cron job polls that mailbox, so this is an on-demand capability gap. See incidents/2026-08-26-work-outlook-basic-auth-disabled.
  • GitHub Trending Radar token claim downgraded to "unverified": the revoked-bot-token (getMe → 401) diagnosis could not be reproduced — c0a9fa314542's last recorded run (2026-09-30 19:22, off-schedule) completed and delivered. Re-confirm at the next scheduled run (Mon 2026-10-05 09:00) before acting on it.

Recent Changes (2026-10-02)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set 3 → 6: two new failures landed on 2026-10-01 —
    • 06a0a1c8704f (AI Newsletter Curation for Patrick, Thu 10:00) — HTTP 401 … All 1 connection(s) authentication expired — please reconnect in the dashboard on provider custom:Omniroute:20128, model cliprox/terra. Its failure notice reached topic 346; no digest was produced. Not self-recovering — needs the OmniRoute connection re-authenticated. Two other enabled jobs share that pin (1b23c042e17d, 79f9bb41ecf5) and will fail next run.
    • 0e753e9720fd (Facebook Birthday Monthly Scrape, 09:00) — HTTP 400, nvidia/deepseek-ai/deepseek-v4-flash absent from the active NVIDIA catalog, confirming the earlier diagnosis that this is a shared-pin problem (sibling fad5a86979d4 uses the same model).
    • 066bb5191280 (Uber Eats Monthly Spend Report, 02:11) — report generated correctly; delivery failed with configured thread_id 34795 for telegram:1793951355 was not found.
  • Carried-over failures, streaks advanced: dca8482e4f76 403 MODEL_NOT_IN_PLAN → streak 6; 1a8e90ea2271 HTTP 410 (retired qwen3.5:397b) → streak 8; fad5a86979d4 HTTP 400 → streak 3. All three still need a model repin.
  • Telegram delivery semantics corrected — a "failed" delivery is not always a lost message. Against ~/.hermes/cron/executions.db, the 2026-10-01 failures carry the fallback variant: configured thread_id <n> for telegram:<dm> was not found; delivered without thread_id. Content reaches the DM, yet the run is recorded delivery_outcome: failed. The 2026-09-30 incident's Message thread not found variant is the one that truly loses the message. Both variants are now documented in systems/messaging-integrations.
  • The wiki has two trees, and they have drifted in both directions (new finding, 2026-10-02): the canonical source is /home/hermes/workspace/agent-estate-wiki — it holds the Gitea remote (Anthony/agent-estate-wiki), the Quartz build (serve.py, package.json, public/) and is what wiki-maintain.sh commits. /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault copy (pages nested under wiki/, no git remote) that the iPad reads. The nightly audit is told to write to the vault path, so edits that skip the canonical tree never reach Gitea or the published site. Worse, the trees have diverged backwards too: systems/ignis.md, systems/nextcloud.md and three incidents/ pages exist only in the vault, while raw/stocktake/*, incidents/2026-09-02-sftpgo-outage.md, reports/communities-wa-social-media-audit-2026.md and agents/ exist only in canonical. index.md in canonical links [[systems/ignis]] and [[systems/nextcloud]], both of which are broken there. Any one-way sync will silently delete pages — reconcile before trusting either tree. See systems/scheduled-tasks.
  • Skills inventory: 344 SKILL.md under ~/.hermes/skills/ (live count 2026-10-02), against 341 in the last generated systems/skills-index (2026-09-27). The page cannot refresh while the stocktake job stays broken.

Recent Changes (2026-10-01)

  • Cron fleet re-verified: 30 jobs, 27 enabled / 3 disabled (live ~/.hermes/cron/jobs.json). Failing set 4 → 3: cbfe434a3b5e returned to ok as predicted on 2026-09-30. The residual 1a8e90ea2271 dispatch error also cleared — its current failure is a different, upstream problem (below).
  • Telegram delivery failures root-caused and fixed: the target, not the topic, was wrong — a bare deliver: "telegram" resolves to the home DM (1793951355), where group topic IDs cannot exist, so sends hard-failed and the message was lost. 1b23c042e17d and c8d52904e1d6 retargeted to telegram:-1004321904721:102 and verified. Three jobs still carry bare telegram targets (0e753e9720fd, fad5a86979d4, 066bb5191280). See systems/messaging-integrations.
  • Three jobs pinned to models that no longer exist upstream — cannot self-recover: 1a8e90ea2271 → qwen3.5:397b retired by Ollama 2026-09-25 (HTTP 410, streak 7; daily 03:00 wiki stocktake is consequently not running); dca8482e4f76 → gpt-5.6-terra plan-gated (403 MODEL_NOT_IN_PLAN, streak 5); fad5a86979d4 → nvidia/…/deepseek-v4-flash absent from the live catalog (HTTP 400, streak 3). Repin to the smart-route alias. See systems/model-providers.
  • GitHub Trending Radar — revoked Telegram bot token: c0a9fa314542 builds its digest then fails the push (getMe → 401). Token in /home/hermes/workspace/github-trending-radar/config.json needs replacing from @BotFather.
  • Memory hygiene pass (2026-09-30): 36 gateway background-process notifications had been written into Mnemosyne as [USER] [Anthony Martin] memories — i.e. machine status text surfacing as the user's own words. Rows purged; canonical facts hermes_cron_model_default and the job-radar model corrected. See systems/mnemosyne.
  • Provider stack rebuilt from live config: 5 custom providers (omni-101, Omniroute:20128, litellm, command-code, guacano). The provider key custom:omni no longer exists; two cron jobs (e530c89f2be0, d40085da631c) still name it but have not failed, because the default custom block resolves to the same base URL.
  • Vision routing — request recorded, not applied: Anthony asked on 2026-09-30 that vision calls use the local Ollama at http://anthonys-mac-mini:11434 (gemma4). Live config still points at nous / stepfun/step-3.7-flash:free as at 2026-10-01.

Recent Changes (2026-09-30)

  • Managed-runtime dependency failure — cron fleet down (2026-09-28 → 30), resolved 19:13 AWST: Jobs failed with Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1). Real cause: the uv-managed runtime venvs (10 under ~/.hermes/installs/019d0d114d22b7e6/environments/*/venv) were missing dotenv and mnemosyne, so the worker died on import before acknowledging the job (ModuleNotFoundError: No module named 'dotenv' — 459 occurrences in worker stderr). Mnemosyne Watchdog failed 176 consecutive runs. Fixed by hermes-runtime-deps-guard.sh v2 (checks/repairs all managed venvs via pip install --target), run every 15 min by job 96eb95ce12f2. weekly-save-restricted-bot-restart script restored 19:17; that job is ok again. See incidents/2026-09-28-hermes-managed-runtime-deps.
  • Hermes upgraded since last audit: v0.19.0 (2026.7.20) → v0.21.5+3837.gbfda74c (2026.9.24), upstream 27062c34, update available. Skills 330 → 341.
  • Provider routing drift: default provider is custom → http://100.78.63.20:20128 (OmniRoute) with default model smart-route; auxiliary curator openai-codex / gpt-5.6-terra. Verified live 2026-09-30.
  • Open — Daily Marketing Job Radar blocked: job dca8482e4f76 pinned to custom:omni-101 / gpt-5.6-terra; upstream command-code returns 403 MODEL_NOT_IN_PLAN. 5 consecutive failures (2026-09-26 → 30). See runbooks/job-radar.
  • Open — Telegram thread delivery failures (root cause found 2026-10-01): telegram:1793951355 thread 34795 "not found" for jobs fad5a86979d4 and 1b23c042e17d; c8d52904e1d6 reports "Message thread not found". Cause was the delivery target, not the topic — a bare telegram resolves to the home DM. Fixed for two of the three jobs on 2026-09-30; see the 2026-10-01 entry above.
  • Wiki tree structure confirmed (drift unresolved): /home/hermes/workspace/agent-estate-wiki is the Gitea/Quartz source (this repo); /home/hermes/wiki/agent-estate-wiki is the Obsidian/Ignis vault (pages under wiki/, no remote). Content is mirrored by hand — the nightly stocktake only maintains this tree.
  • Multi-bot group chat question raised: Anthony asked whether other estate Hermes instances can share a group chat. Preferred shape is a Telegram supergroup with one forum topic per bot (Mattermost rejected). Design question only — nothing deployed.

Recent Changes (2026-09-02)

  • SFTPGo outage resolved (02:18 AWST): Dashboard flagged "service down". CT299 swap fully exhausted (43Gi used, 100%); CT299's tailscaled had died, breaking tailnet reachability; PVE's pct start then failed because 5 stale lxc-attach probes from earlier diagnostics pinned CT299's cgroup (/sys/fs/cgroup/lxc/299/ns/.lxc). Killed those PIDs (3037951/3334928/3352203/3402511/4004148), pct stop 299 --skiplock + pct start 299, fresh PID 3471141, tailscaled reconnected direct. Verified end-to-end: tailscale ping sftpgo → direct 192.168.178.92:41641, SFTPGo WebAdmin HTTP 401, SFTP banner SSH-2.0-SFTPGo_2.7.0. See incidents/2026-09-02-sftpgo-outage.

Attested facts

  • Config version: _config_version: 46 — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously recorded as 33 on 2026-07-25)
  • Default profile: default — Verified, WebUI prompt, 2026-07-22
  • Timezone: Australia/Perth — User-provided + Verified in config, 2026-07-22
  • Main provider: custom → http://100.78.63.20:20128 (OmniRoute) — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously custom:litellm → litellm:4000, then custom:omni → omni.kangaroo-eel.ts.net)
  • Default model: smart-route — Verified, ~/.hermes/config.yaml, 2026-09-30 (previously freellmapi-auto)
  • Fallback providers: opencode-go / mimo-v2.5, opencode-zen / deepseek-v4-flash-free, nous / stepfun/step-3.7-flash:free, ollama-cloud / qwen3.5:397b, kilocode / poolside/laguna-m.1:free, custom / litellm:4000 — Verified, config, 2026-09-30
  • Auxiliary curator model: openai-codex / gpt-5.6-terra — Verified, config, 2026-09-30
  • Memory backend: Mnemosyne — Verified, config, 2026-07-22
  • Memory enabled: true — Verified, config, 2026-07-22
  • Streaming enabled: true — Verified, config, 2026-07-22
  • Dashboard public URL: https://hermes.martinwa.org/ — Verified, config, 2026-07-22
  • Pet config: enabled/slug null — Verified, config, 2026-07-22
  • WebUI connected platforms: local, api_server, discord, email, homeassistant, mattermost, ntfy, slack, telegram, webhook — Verified, WebUI session, 2026-07-22

Recent Changes (2026-08-02)

  • Browser backend reverted: Camoufox restored as primary on 2026-07-30. Steel Browser (CT201) experiment ended — CDP issues + DeepSeek 0731 tool name validation. browser.cloud_provider: camofox, camera_provider: camofox. Steel plugin disabled (the 2026-08-02 state — the plugin and the Steel deployments were later removed entirely on 2026-10-29, see systems/browser-backend). See comparisons/steel-vs-camoufox and decisions/rollback-steel-to-camofox.
  • Mnemosyne capture fix (2026-08-01): Automatic conversational capture was broken by stringified YAML (sync_roles: "['user']"). Fixed by rewriting structured Mnemosyne settings with native YAML types. See incidents/2026-08-01-mnemosyne-capture-failure.
  • Mnemosyne vector coverage gap discovered: Only 159/1,476 working memories (10.8%) have vector embeddings. 0/458 episodic memories have embeddings. Semantic search is mostly FTS5 fallback.
  • DeepSeek V4 Flash 0731: Post-training bump, same model ID, same pricing. Terminal Bench 2.1: 82.7 (up from 61.8). Stricter tool name validation (^[a-zA-Z0-9_-]+$) — may cause issues with OmniRoute-transformed tool names.
  • Uber Eats report upgraded: Now uses Camofox authenticated session (not stale cookies.txt). HTML + wiki report generated. Cron job updated with camofox-browser-automation + browser-backend-verification skills.
  • GitHub Trending Radar — Daily Digest removed: Stopped by user on 2026-08-02. Weekly roundup still active.
  • Ops profile renamed: Tembo → Tusk. All SOUL.md, skill references, and memory entries updated.
  • Wiki audit cron job created: Runs nightly at 23:59 AWST — reviews past 24h of conversation for wiki changes.
  • Ops profile Mnemosyne enabled: Mnemosyne plugin enabled on ops profile with profile_isolation: true — Tusk gets its own memory bank, separate from default (Rhino). Zero cross-contamination. Verified end-to-end 2026-08-02. Config: ~/.hermes/profiles/ops/config.yaml (memory.mnemosyne.profile_isolation: true, allow_tool_override: false).
  • OmniRoute free-DeepSeek silent-stop mitigation (2026-08-04): Free-DeepSeek upstreams returning empty-after-tool responses since Aug 1 → silent turn stops on deepseek-flash-free and smart-route/good-free across both instances. Reworked both combos: strategy priority → fill-first, dropped proven culprits (238d5978/deepseek-v4-flash + -0731, nvidia deepseek), healthy free models promoted, deepseek demoted to last resort. See incidents/2026-08-04-omniroute-free-deepseek-silent-stops.
  • OmniRoute three-layer fix (2026-08-07): npm package .env override (ENABLE_SOCKS5_PROXY=true re-installed by npm update), dead nanobot provider connection (CT333 DOWN), PVE host load leaking into CT600 via nesting=1. First-request-after-cache-expiry stall documented. See systems/omniroute.
  • Nanobot status degraded (2026-08-07): CT333 nanobot confirmed DOWN — connection refused on 8900. OmniRoute provider connection disabled. See systems/nanobot.
  • CT460 memory bumped (2026-08-07): 5096MB → 14GB RAM. Was causing swap thrashing at load 8+.

Recent Changes (2026-08-08)

  • Cron job model repinning: 3 jobs repinned from dead freeapi/minimax-m2.7 → smart-route (IDs: c8d52904e1d6, 1b23c042e17d, d40085da631c). Mosquito Tracker repinned from dead freeapi/auto → smart-route (ID: 1b31187397a5). 9 stale-named cron jobs (e.g. vertex/gemini-2.5-flash, gpt-5.6-terra, deepseek-v4-flash-free) repinned to smart-route. Daily Marketing Job Radar swapped to gpt-5.6-terra.
  • Hermes UI vite config revert: Vite config had been changed to HTTPS with self-signed certs (~16:00 Aug 8), breaking iPad access (plain HTTP → HTTPS-only without trusted cert). Reverted to pre-HTTPS config, restarted hermes-ui.service. Production build now serves /ui/ path correctly with relative asset paths. hermes-desktop.martinwa.org identified as separate Cloudflare infrastructure (likely on CT450), not fixable from CT460.
  • Phase One estate review completed: Manual cron run verified CT450's 11 duplicate publishing jobs remain paused (zero executions since Aug 1), CT460's 11 production jobs remain enabled with last_status=ok, and CT333 Nanobot has a live jobs.json with 9 jobs. The cron review report contained two inaccuracies (wrong date, false claim about CT333 having no scheduler) — verified via live system probes instead.

Recent Changes (2026-08-09)

  • Portainer password reset (CT245): Admin user is anthony (not admin). Password was forgotten; reset via docker run --rm -v portainer_data:/data portainer/helper-reset-password after stopping the Portainer container. Verified end-to-end with POST /api/auth → HTTP 200 + valid JWT. Portainer now uses Tailscale/Let's Encrypt cert on :9443 (see systems/homepage-dashboard). Recovery procedure: stop container → run helper image → restart → verify with auth API.
  • SFTPGo security review (CT299): Anthony shared SFTPGo WebAdmin PDF settings for audit. Key findings: (1) remote SFTP root path // resolves to / — full filesystem access, (2) host fingerprints empty (low-risk on Tailscale but should be pinned for hygiene), (3) remote user is root, (4) quota value F1 on Cloudflare-R2 virtual folder is non-standard (may be display artifact). No action taken — flagged for Anthony's review.

Recent Changes (2026-08-10)

  • Cron job failures — session storage issue: Four cron jobs failed on 2026-08-10 with two distinct error patterns:
    • TimeoutError (idle for 601–602s): "Daily Marketing Job Radar" (dca8482e4f76) at 07:12, "Political News Digest (Mon/Thu)" (0e519874d280) at 08:12. Jobs stuck initializing, exceeded 600s idle limit.
    • RuntimeError (session storage): "Tech & AI Newsletter Digest (Analytical)" (85edf6772bca) at 07:12, "GitHub Trending Radar — Weekly Roundup" (c0a9fa314542) at 09:03. Error: "session storage could not be written (the transcript would have been lost on restart). This is often a full disk — free some space (or fix state.db permissions)."
    • Root cause: Disk at 83% (11GB free on 65GB). state.db is 593MB. Not critically full but may be triggering write failures. Sessions.json shows only 1 active session in past 24h — session DB likely needs cleanup or WAL checkpoint.
    • Impact: News digests and GitHub Trending did not deliver. Job Radar did not run.
    • See: systems/scheduled-tasks Known Issues section.
  • OpenCode WebSocket proxy errors: Repeated "OpenCode service unavailable" errors in hermes node journal (Aug 11 00:08). Process running (PID 809564, opencode serve), but WebSocket proxy failing to connect. Likely transient — service was available earlier in the day.

Recent Changes (2026-08-14)

  • OmniRoute factory reset on CT601: Database wiped clean (0 combos, 0 providers, 0 API keys, 115 built-in auto entries). Backup at /opt/omniroute/factory-reset-backup-20260814-225524. Providers re-added from Notion API vault: Anthropic, Google Gemini, Groq, Mistral, OpenRouter (all passed live tests). OpenAI keys tested but invalid — removed. See systems/omniroute.
  • Hermes default model changed: auto/best-free → resilient-free in ~/.hermes/config.yaml. auto/best-free was misconfigured (no combo definition, routed to paid gpt-5.6-sol).
  • Multiple free combos broken by upstream model ID changes: OpenRouter Poolside laguna-s-2-1:free → laguna-s-2.1:free (dash→dot); Groq llama-4-scout-17b-16e-instruct retired. Affected: resilient-free, free-IA, free-stack, good-free, deepseek-flash-free, mimo-free.
  • Guanaco router updated: CT205 ollama container — Guanaco v0.8.11 → v0.8.14. DSML tag leakage fixes (tool-call formatting no longer leaks raw syntax), dashboard performance fix. See systems/guanaco.
  • GPT-5.6 Luna research: Community consensus — Luna cleaner/polished; DeepSeek V4 Flash more persistent/cheaper. Provider route matters more than model name. Luna recommended for planning/review, DeepSeek for execution. See systems/model-providers.
  • Hermes UI restart: Port 8787 service restarted, now properly binding 0.0.0.0 for external access. Confirmed reachable from Tailscale.
  • LiteLLM routing issue identified: Hermes was previously pointed at LiteLLM (http://litellm:4000/v1, model free-auto) causing 6–26s per call. OmniRoute auto/best-free tested at 265ms. Config now points at OmniRoute.

Recent Changes (2026-08-13)

  • OmniRoute migrated CT600 → CT601: Provider renamed custom:omniroute → custom:omni; base URL http://omniroute:20128/v1 → http://omni:20128/v1. Host omni.kangaroo-eel.ts.net (100.93.204.4, active) replaces omniroute (100.88.81.19, offline). CT600 stopped, CT601 running. DB path /var/lib/omniroute/storage.sqlite → /opt/omniroute/storage.sqlite; API key unchanged. All config (14) + cron-job (22) custom:omniroute refs repointed to custom:omni. Backup: ~/.hermes/config.yaml.bak.omniroute-removal. See systems/omniroute.
  • Evening stand-down briefing weather bugfix: Weather collector returns a plain string (+16°C Sunny), but the three dashboard widgets (evening.html, evening-grid.html, evening-briefing.html) read object fields → undefined. Added string-vs-object guards; verified served HTML no longer contains broken raw field refs.